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ABSTRACT 


Human  systems  integration  (HSI)  applies  knowledge  of  human  capabilities  and 
limitations  to  design  more  efficient,  effective  and  safe  military  systems.  HSI  can 
enable  impressive  lifecycle  cost  savings  and  performance  gains  when 
implemented.  In  practice,  HSI  activities  are  hampered  by  the  complexity  of 
human-technology  integration  issues.  This  thesis  develops  a  simplified 
framework  for  understanding  and  assessing  HSI  throughout  the  acquisition 
process. 

An  HSI  Activity  Model  is  presented  to  conceptualize  HSI  activity  in  military 
acquisition.  Established  human  factors  and  human  computer  interaction  theories 
are  applied  to  develop  a  concise  view  of  HSI  in  action.  The  core  activity  of  HSI  is 
summarized  as  the  balancing  of  human  capabilities  and  limitations  with  the 
affordances  and  constraints  presented  by  system  technology,  to  accomplish 
system  objectives. 

A  Comprehensive  Human  Integration  Evaluation  Framework  (CHIEF)  is 
then  developed  to  provide  the  acquisition  community  with  a  viable  tool  for 
assessing  HSI  during  acquisition.  A  measurement  approach,  rating  scales  and 
criteria,  and  a  consolidated  scoring  matrix  are  developed  based  on  lessons 
gathered  from  current  system  assessment  measures,  and  refinement  with  HSI 
practitioners. 

If  implemented,  the  HSI  Activity  Model  and  CHIEF  offer  the  potential  to 
increase  HSI  understanding  and  awareness,  leading  to  improved  system 
outcomes  across  the  DHS  acquisition  enterprise. 
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EXECUTIVE  SUMMARY 


Human  systems  integration  (HSI)  incorporates  knowledge  of  human  capabilities 
and  limitations  into  the  design  of  modern  systems  to  make  them  more  efficient, 
effective  and  safe  (Booher,  2003).  When  implemented  in  a  timely  fashion,  HSI 
can  drive  impressive  cost  savings  and  performance  gains  (Booher,  1997;  Liu, 
Valerdi,  &  Rhodes  2009).  In  application,  HSI  is  hampered  by  the  complexity  of 
human-technology  integration  issues  (Newman,  Bruseberg,  Lowe,  Borras,  & 
Tatlock,  2008;  Boff,  1990).  This  thesis  presents  two  products  to  enhance 
understanding  of  HSI  during  military  acquisition:  a  conceptual  model,  and  an 
assessment  framework. 

The  HSI  activity  model  conceptualizes  HSI  activity  during  military 
acquisition.  Established  human  factors  and  human-computer  interaction  theories 
were  applied  to  military  acquisition  to  develop  a  concise  view  of  HSI  in  action. 
The  model  summarizes  the  core  activity  of  HSI  as  the  balancing  of  human 
capabilities  and  limitations  with  the  affordances  and  constraints  presented  by 
system  technology  to  accomplish  system  objectives  (Figure  1). 


Figure  1.  Proposed  HSI  activity  model. 


xv 


The  Comprehensive  Human  Integration  Evaluation  Framework  (CHIEF) 
was  developed  to  provide  the  acquisition  community  with  a  viable  tool  for 
assessing  HSI  during  acquisition.  Systems  assessment  measures,  including 
NASA/DOD  technology  readiness  levels  (TRL),  were  analyzed  to  determine 
criteria  for  a  successful  measure.  The  measurement  approach,  rating  scales  and 
criteria,  and  a  consolidated  scoring  matrix  were  developed  in  collaboration  with 
faculty  from  the  Naval  Postgraduate  School.  Elements  of  CHIEF  were  then 
refined  during  a  series  of  workshops  with  practitioners  from  the  Coast  Guard 
Human  Systems  Integration  Division  (CG-1B3).  As  a  culminating  exercise,  the 
practitioners  evaluated  the  usability  of  CHIEF  by  conducting  a  mock  HSI 
evaluation  for  a  current  major  acquisition  program.  The  finalized  CHIEF 
framework  elements  are  presented  in  Figure  2. 
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Figure  2. Comprehensive  Human  Evaluation  Framework  (CHIEF)  elements 
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Areas  for  expanded  investment  and  research  include  the  development  of 
dedicated  CHIEF  software,  identification  of  measures  to  support  the  framework, 
and  refinement  of  rating  scales  and  criteria  based  on  organizational  needs.  If 
implemented,  the  HSI  Activity  Model  and  CHIEF  offer  the  potential  to  increase 
HSI  understanding  and  awareness  during  the  acquisition  process,  leading  to 
improved  system  outcomes  across  the  DHS  acquisition  enterprise. 
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I.  INTRODUCTION 


A.  WHAT  IS  HUMAN  SYSTEMS  INTEGRATION? 

Human  systems  integration  (HSI)  is  an  amalgam  of  many  fields,  unified  by 
a  pragmatic  purpose:  to  understand  human  capabilities  and  limitations  and  apply 
this  knowledge  to  system  design  (Chapanis,  1996;  Booher,  2003).  HSI  has  been 
described  as  a  management  concept,  a  design  philosophy,  and  a  technical 
approach  centered  on  human  users  of  modern  systems  (Booher,  2003).  The 
International  Council  on  Systems  Engineering  (INCOSE)  Handbook  captures  this 
purpose:  “The  primary  objective  of  HSI  is  to  ensure  that  human  capabilities  and 
limitations  are  treated  as  a  critical  system  element,  regardless  of  whether 
humans  in  the  system  operate  as  individuals,  crews,  teams,  units,  or 
organizations.”  (Haskins,  Forsberg,  Krueger,  Walden,  &  Hamelin,  2006,  p.  326) 

HSI  is  deeply  rooted  in  the  field  of  human  factors  (HF),  originating  from 
studies  of  task  efficiency  among  factory  workers  in  the  early  1900s.  By  mid¬ 
century,  research  in  human  factors  engineering  (HFE)  enabled  better 
understanding  of  how  man  interacts  with  technology  (Figure  1).  This  knowledge 
was  integrated  into  the  increasingly  complex  technological  systems  that  emerged 
from  the  Second  World  War  (Chapanis,  1996;  Tvaryanas,  2010). 


Figure  1.  Human  factors  during  WWII  (from  DVI  Aviation,  Inc) 
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Our  expanding  knowledge  of  human-technology  interaction  led  to  more 
diverse  approaches  in  the  latter  half  of  the  20th  century.  New  ways  to  integrate 
human  considerations  into  systems  development  were  conceived,  and  areas  of 
research  were  expanded  to  involve  multiple  domains.  Rather  than  focusing 
narrowly  on  individual  workstation  or  task  design,  practitioners  addressed  staffing 
policies,  training  approaches,  and  the  effects  of  health  and  safety  on  overall 
system  performance  (Chapanis,  1996,  Tvaryanas,  2010).  These  considerations 
became  a  key  factor  in  the  evaluation  of  system  performance.  With  the  advent  of 
the  U.S.  Army’s  MANPRINT  program  in  the  late  1980s,  HSI  was  incorporated 
into  modern  military  system  acquisition  (Booher,  2003,  Tvaryanas,  2010). 

Organizations  across  the  DOD  and  federal  government  recognize  different 
domains  within  the  HSI  discipline.  This  thesis  adopts  the  seven  domains 
recognized  by  the  U.S.  Coast  Guard:  human  factors  engineering,  personnel, 
manpower,  performance  support  and  training,  systems  safety  and  occupational 
health,  survivability,  and  habitability  (Figure  2). 


Human  Factors  Engineering  (HFE):  Employed  during  systems  engineering  over 
the  life  of  the  program  to  provide  for  effective  human-machine  interfaces  and  to 
meet  HSI  requirements. 

Personnel:  Define  the  human  performance  characteristics  of  the  user  population 
based  on  die  system  description  and  projected  characteristics  of  target  occupational 
specialties.  Per  sonnel  attributes  are  design  parameters 

Manpower-:  The  mix  of  military,  civilian,  and  contract  support  necessary  to 
operate,  maintain,  train  and  support  the  system. 

Performance  Support  and  Training  (PS&T):  Develops  options  for  individual, 
collective,  and  joint  training  for  operator  s,  maintainers  and  support  personnel, 
consistent  with  FORCECOM  policies  and,  where  appropriate,  base  training 
decisions  on  gaining  effectiv  eness  evaluations.  Tire  PM  shall  address  the  major 
elements  of  training,  and  place  special  emphasis  on  options  that  enhance  user 
capabilities,  maintain  skill  proficiencies,  and  reduce  individual  and  collective 
training  costs. 

System  Safety  and  Occupational  Health  (SS/OH):  This  domain  integrates 
across  disciplines  and  into  systems  engineering  to  determine  system  design 
characteristics  that  can  minimize  the  risks  of  acute  or  chronic  illness,  disability,  or 
death  or  injury  to  operators  and  maintainers;  and  of  equipment  damage,  failure  or 

loss. 

Survivability:  Addresses  personnel  survivability  issues  including  protection 
against  detection,  fratricide.  Chemical,  Biological.  Nuclear.  Radiation  and 
High-Yield  Explosives  (CBNRE)  effects;  the  integr  ity  of  the  crew  compartment; 
and  provisions  for-  rapid  egress. 

Habitability:  Establishes  requirements  for  the  physical  environment,  personnel 
services  (e.g.,  medical  and  messing),  working  and  living  conditions  (e.g. .  berthing 
and  personal  hygiene). 


Figure  2.  USCG  HSI  domains  (from  Acquisition  Directorate,  2010) 
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HSI  is  not  a  collection  of  discrete  disciplines,  but  various  domains  sharing 
a  set  of  concepts  and  similar  language.  In  practice,  the  borders  between  domains 
are  porous  and  their  contents  are  interrelated  (Booher,  2003).  The  labels 
differentiating  domains  are  mainly  a  convenient  taxonomy  that  suggests  the 
perspectives  from  which  human-centered  design  issues  may  be  understood. 

B.  HSI  IN  PRACTICE 

HSI  contains  a  rich  body  of  knowledge  spanning  more  than  a  century  of 
research  in  the  social,  physical,  and  cognitive  sciences.  HSI  and  HF  research 
offers  empirically-grounded  understanding  of  human  capabilities  and  limitations, 
and  time-tested  sociotechnical  design  principles  to  the  acquisitions  community  of 
practice  (Chapanis,  1996).  HSI  Practitioners  employ  this  specialized  knowledge 
with  a  robust  suite  of  tools,  techniques,  and  methods  (Hale,  Ching,  Brett,  & 
Rothblum,  2009)  to  investigate  the  integration  of  man  and  machine  in  modern 
systems. 

HSI  practitioners  work  in  integrated  product  teams  (IPTs)  across  every 
phase  of  military  acquisition  (USCG  Acquisition  Directorate,  2010).  They 
contribute  early  by  providing  input  to  mission  need  statements  (MNSs)  and 
operational  requirements  documents  (ORDs).  As  a  system’s  functional  and 
physical  architecture  begins  to  take  shape,  practitioners  work  across  functional 
areas  to  incorporate  human-centered  design  principles.  In  later  stages, 
practitioners  facilitate  human-centered  program  management,  contracting,  and 
test  and  evaluation  (T&E)  activities. 

HSI  practitioners  come  from  diverse  disciplines  and  professional 
backgrounds.  Qualifications  range  from  graduate  and  undergraduate  degrees  in 
the  human  sciences  to  professional  certifications  (Kleiner  &  Booher,  2003; 
Chapanis,  1996).  While  their  backgrounds  and  expertise  vary,  HSI  practitioners 
occupy  a  distinctive  role  in  acquisition.  They  advocate  for  system  operators, 
maintainers,  and  supporters  by  ensuring  that  systems  attributes  are  designed  to 
support  their  needs. 
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c. 


HSI  REALIZATIONS 


Decades  of  HF  and  HSI  have  revealed  a  number  of  important  realities 
concerning  human-centered  design.  This  section  addresses  several  critical 
themes. 

1.  It  Is  Hard  to  Get  HSI  Right 

Human  beings  are  deceivingly  complex,  and  the  questions  that  arise  in 
intermingling  people  with  technology  warrant  very  careful  study.  Anthropometry 
(the  science  of  measuring  the  parameters  of  the  human  body)  is  but  one 
example  of  human  complexity.  Properly  derived  anthropometric  design  ensures 
that  a  system  will  physically  accommodate  its  intended  users  (Pheasant  & 
Haslegrave,  2005).  Anthropometric  conformance  is  often  addressed  by  applying 
the  “5th-95th  percentile”  criterion  (Blanchard  &  Fabrycky,  2011;  NASA,  2010). 
This  criterion  mandates  that  a  system  accommodate  all  users  that  fall  within  the 
5th  to  95th  percentile  (central  90  percent)  for  relevant  physical  measurements.  In 
commercial  application,  the  use  of  anthropometries  is  seen  in  the  adjustability  of 
driver  seats  to  ensure  that  a  wide  range  of  customers  can  operate  a  vehicle’s 
controls  effectively.  A  sampling  of  anthropometric  measures  common  to 
commercial  and  military  applications  is  shown  in  Figure  3. 
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No. 

Dimension 

Min  (cm.  (in)) 

Max  (cm.  (in)) 

7$8 

Sitting  height 

777(30  6) 

101  3(39  9) 

330 

Eye  height,  sitting 

66  5  (26.2) 

88  9  (35.0) 

529 

Knee  height,  sittmg 

45.5(17  9) 

63.5(25.0) 

678 

Popliteal  height 

33  0(13.0) 

50.0  (19  7) 

751 

Shoulder>elbow  length 

29  6(11.6) 

41  9(16  5) 

194 

Buttock -knee  length 

52  1  (20  5) 

69.9  (27.5) 

Figure  3.  Basic  anthropometric  measures  (from  NASA,  2010) 
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Accounting  for  height,  reach,  leg  length  and  other  body  measurements  in 
design  might  appear  to  be  a  simple  matter  of  data  retrieval:  find  the  applicable 
database  (as  in  Figure  3),  select  the  measurement  (e.g.,  arm  length),  and  design 
to  accommodate  the  desired  range  of  measures.  In  most  cases  a  significantly 
more  detailed  anthropometric  analysis  must  be  applied.  A  person  with  long  legs, 
for  instance,  will  not  necessarily  have  a  long  torso,  or  long  arms,  as  Figure  4 
illustrates. 


Figure  4.  Differences  between  measures  for  two  individuals  with 
identical  seated  height  (from  McCauley,  2014) 


Humans  are  not  constructed  with  uniform  proportions  (Pheasant  & 
Haslegrave,  2005).  Humans  are  in  fact  so  variable  that  the  differences  seen  in 
Figure  4  are  common  across  nearly  every  physical  measure  (Moroney  &  Smith, 
1972).  Because  human  variability  is  so  pervasive,  simplistic  approaches  are 
often  problematic.  The  development  of  21st  century  military  systems  has 
confirmed  that  a  refined,  multivariate  analysis  is  required  to  calibrate  systems  to 
users  (Lockett,  Kozycki,  Gordon,  Bellandi,  2005).  Multivariate  analysis  (Figure  5) 
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accounts  for  all  potential  combinations  of  individual  anthropometric  measures, 
using  specialized  modeling  software  (Lockett  et  al.,  2005;  NASA,  2010). 


Figure  5.  Reach  envelopes  calculated  for  different  users 
(from  Ergo-Link  by  Sun  Group  Design,  LLC  ) 

Consider  that  a  task  as  simple  as  pressing  a  button  on  a  console  involves 
more  than  ten  anthropometric  measures  (Pheasant  &  Haslegrave,  2005)  and 
countless  potential  paths  of  movement.  When  the  complexity  of  this  physical  task 
is  coupled  with  the  complexity  of  the  cognitive  processes  that  guide  human 
movement,  the  challenging  nature  of  designing  human-technology  interfaces 
becomes  apparent. 

2.  HSI  Is  Inevitable 

Jakob  Nielsen,  co-founder  of  the  Nielsen-Norman  consulting  group  and 
former  vice  president  of  research  at  Apple  Computer,  once  had  this  to  say 
regarding  the  testing  of  a  new  product: 

your  system  will  be  tested  for  usability  even  if  you  don't  do  so 

yourself.  Your  customers  will  do  it  for  you,  as  they  struggle  to  use 
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the  system.  Any  usability  problems  found  by  users  in  the  field  will 
undermine  your  reputation  for  quality  products  and  the  resulting 
change  requests  will  be  about  100  times  more  expensive  to 
implement  than  changes  discovered  by  yourself  in  the  early  phases 
of  the  project.  (Nielsen,  1993,  pp.  7-8) 

The  true  customers  in  a  military  acquisition  are  military  users.  It  is  these 
operators,  maintainers,  and  supporters  who  eventually  transform  an  assembly  of 
technologies  we  know  as  a  “system”  into  a  “capability”  (Booher,  2003).  Even  in 
the  case  of  unmanned  or  autonomous  systems,  living  users  contribute  directly  or 
indirectly  to  almost  every  function  (Murphy  and  Shields,  2012).  Like  users  of 
commercial  products,  military  users  determine  a  system’s  success  or  failure. 
However,  unlike  civilian  consumers,  military  users  often  have  less  discretion  in 
selecting  the  equipment  they  use  to  complete  their  tasks. 

In  the  consumer  world,  a  system  poorly  calibrated  to  a  customer's 
capabilities  and  limitations  may  lead  to  poor  product  ratings  and  limp  sales 
(Nielsen,  1993).  But  poor  HSI  in  a  military  system  may  bring  error,  mishap,  and 
injury  that  may  otherwise  have  been  avoided  (Booher,  1997;  Tvaryanas,  2006). 
Beyond  the  vicissitudes  of  corporate  fortunes,  mission  outcomes,  and  thus 
human  lives,  hang  in  the  balance.  The  nature  of  military  HSI  demands  that  we 
identify  and  remove  system  performance  detractors  to  the  full  extent  of  system 
resources. 

Systems  eventually  end  up  in  the  hands  of  users.  Poor  design  elements 
are  exposed,  and  changes  will  inevitably  result.  The  lesson  for  military  systems  is 
straightforward:  changes  will  always  be  necessary  to  make  products  suitable  for 
operational  use;  it  becomes  a  simple  matter  of  how  much  we  want  to  pay. 

3.  HSI’s  Timing  Is  Critical 

As  the  Coast  Guard  and  DHS  develop  increasingly  complex  systems 
under  tighter  fiscal  and  manpower  constraints,  timely  and  effective  HSI 
involvement  grows  increasingly  important  (Boff,  1990;  Chapanis,  1996;  Booher, 
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2003).  Solving  problems  in  human-technology  integration  is  easily  deferred  in 
the  immediacy  of  ensuring  that  system  technology  is  being  developed  as 
scheduled  and  within  cost  constraints.  While  meeting  deadlines  and  schedule 
constraints  may  be  hallmarks  of  a  well-executed  acquisition,  early  identification 
and  implementation  of  HSI-driven  design  improvements  are  also  critical  to  a 
program’s  ultimate  success  (Booher,  2003,  Miller  et.  al.,  2003). 
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Figure  2-4  Committed  Life-cycle  Cost  against  Time1 

Figure  6.  The  cost  of  system  changes  over  time 
(from  the  INCOSE  handbook) 


As  seen  in  Figure  6,  postponing  HSI  has  a  price  tag.  The  costs  associated 
with  delaying  HSI-related  design  changes  increase  quickly  as  a  system  matures 
(Blanchard  &  Fabrycky,  2011;  Haskins  et  al.,  2006).  For  example,  it  may  become 
apparent  that  the  operating  console  on  the  bridge  of  a  new  Coast  Guard  cutter 
should  be  relocated  to  improve  watch  awareness  and  maintenance  access.  In 
the  early  stages  of  acquisition,  the  console  can  be  relocated  with  mouse-clicks  in 
the  design  software.  If  the  change  is  delayed  until  production  has  begun,  a  formal 
engineering  change  must  be  initiated,  requiring  extensive  management  and 
contracting  overhead,  in  addition  to  material  and  labor  costs.  Figure  7,  adopted 
from  Systems  Engineering  &  Analysis,  5th  edition,  illustrates  this  point. 


8 


Figure  7.  HSI  cost  effectiveness  over  time 
(after  Blanchard  &  Fabrycky,  2011) 


HSI  makes  systems  better  by  ensuring  that  technological  systems  marry 
well  with  the  realities  of  their  human  users.  Designing  effective,  efficient,  and  safe 
systems  for  our  service  members  requires  careful,  comprehensive,  and  timely 
human-technology  integration.  As  commercial  industry  teaches  us,  changes  are 
inevitable  as  human  and  technological  elements  of  a  system  come  together. 
What  is  not  inevitable,  however,  is  incurring  excess  cost  because  of  these 
changes.  Deferring  HSI  to  the  late  stages  of  acquisition  is  enormously  expensive. 
In  military  acquisition,  the  choice  is  ours:  how  much  will  we  ask  taxpayers  to  pay? 

D.  THE  IMPETUS  FOR  THIS  THESIS 

The  Human  Factors  and  Behavioral  Sciences  Division  of  the  Department 
of  Homeland  Security  (DHS)  published  “A  Vision  for  Human  Systems  Integration 
in  the  U.S.  Department  of  Homeland  Security”  in  2009.  The  article  identified  ten 
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keys  to  implementing  HSI  within  DHS.  First  among  these  was  the  following: 
“Standardize  and  institutionalize  the  process  for  embedding  HSI  into  the  DHS 
Technology  Readiness  Level  [TRL]  framework”  (Wilson,  Malone,  Lockett- 
Reynolds,  &  Wilson,  2009,  p.  2).” 

DHS’s  emphasis  on  integrating  human  concerns  into  technological 
development  is  not  surprising.  In  the  next  chapter,  our  study  of  the  NASA/DOD 
TRL  scale  will  reveal  a  framework  that  enables  technology  to  be  managed  across 
diverse  acquisition  disciplines  in  a  timely  and  effective  manner.  By  contrast,  few 
frameworks  exist  for  evaluating  how  users  integrate  with  technology  (Johnston  et 
al. ,  2013;  Phillips,  2010).  Stated  plainly,  human-integration  considerations  are  at 
a  severe  disadvantage  in  the  acquisition  process,  as  compared  to  technologically 
driven  requirements. 

The  USCG  and  DHS  are  not  the  only  organizations  faced  with  this 
challenge.  Implementing  HSI  and  HF  effectively  is  a  mounting  concern  within  the 
Department  of  Defense  (DOD)  and  other  federal  agencies,  the  U.K.’s  ministry  of 
defense  (MoD),  and  the  commercial  sector  (Johnston  et  al.,  2013;  Earthy  et  al., 
1999).  Although  developmental  measures  have  been  proposed,  there  is  no 
widely  accepted  framework  for  evaluating  HSI  throughout  the  acquisition 
lifecycle. 

This  thesis  proposes  a  conceptual  model  of  the  core  activity  of  HSI  and  a 
framework  for  assessing  HSI  during  acquisition.  If  successfully  implemented, 
these  products  are  expected  to  illuminate  human-technology  integration  issues, 
allowing  improved  outcomes  across  the  USCG/DHS  acquisitions  enterprise. 

The  next  chapter  examines  the  development  and  emergence  of  the 
NASA/DOD  TRL  scale,  which  is  widely  used  for  technology  readiness 
assessment  in  federal  acquisition.  Recently  proposed  human-technology 
integration  measures  are  also  examined  and  compared  with  TRL  to  establish 
criteria  for  an  HSI  assessment  framework. 
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II.  A  REVIEW  OF  CURRENT  SYSTEM  READINESS  MEASURES 


While  a  number  of  measures  have  been  developed  to  assess  system 
readiness,  assessing  human-technology  integration  is  a  relatively  new 
undertaking  (Phillips,  2010;  Johnston  et  al.,  2013).  In  this  section,  the 
NASA/DOD  TRL  scale  is  analyzed  and  compared  with  two  developmental 
human-technology  assessment  frameworks  (SHARE  and  HRL).  Important  links 
between  these  assessment  frameworks  and  the  TRL  scale  are  noted  and  the 
drawbacks  of  using  the  scale’s  structure  to  assess  HF  and  HSI  are  identified. 

A.  LESSONS  FROM  THE  EMERGENCE  OF  NASA/DOD  TECHNOLOGY 

READINESS  LEVEL  SCALE 

The  NASA/DOD  TRL  scale  is  now  widely  accepted  in  federal  acquisition 
(Johnston  et  al.,  2013,  Mankins  2009).  This  scale  focuses  exclusively  on 
technology,  and  makes  no  attempt  to  account  for  human  contributions  to  a 
system  (Phillips,  2010).  Nevertheless,  its  conception  and  successful  propagation 
throughout  the  acquisition  community  offers  lessons  for  establishing  a  viable 
systems  assessment  measure.  To  understand  the  importance  of  the  TRL  scale, 
some  historical  context  is  required. 

During  America’s  quest  to  dominate  space  in  the  1970s,  NASA  was 
charged  with  integrating  cutting-edge  technology  into  spacecraft  at  an 
extraordinary  rate.  At  the  time,  technology  readiness  assessment  (TRA)  was  the 
chief  methodology  for  ensuring  technological  components  were  suitably  mature 
for  NASA  missions  (Mankins,  2009).  Assessments  were  conducted  at  multiple 
points  in  the  development  lifecycle,  requiring  detailed  technical  requirements  to 
be  translated  across  government,  commercial,  and  scientific  circles.  The  need  to 
develop  a  unified  standard  for  technology  readiness  soon  became  evident,  and 
what  we  now  know  as  the  TRL  scale  emerged  (Mankins,  1995;  Mankins,  2009). 

The  current  TRL  scale  divides  technological  readiness  into  nine  simply 
defined  levels.  The  scale  produces  “a  discipline-independent,  program  figure  of 
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merit”  (Mankins,  2009,  p.  1216)  with  which  to  assess  and  communicate  the 
maturity  of  a  new  technology.  This  allows  complex  technology  readiness 
requirements  to  be  understood  and  applied  across  disciplines.  In  a  recent 
retrospective,  John  Mankins,  a  25-year  veteran  at  NASA,  highlighted  the  TRL 
scale  as  “the  foundation  of  modern  technology  readiness  assessment”  (Mankins, 
2009,  p.  1217).  This  foundation  has  shaped  NASA  technology  development  for 
more  than  three  decades.  The  current  version  of  the  TRL  scale  appears  in  Figure 
8. 
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Figure  8.  The  NASA  TRL  scale  (from  NASA,  2012) 


More  than  three  decades  after  implementation,  users  of  the  TRL  scale 
include  DHS,  USCG,  DOD,  and  government  organizations  throughout  the  world 
(Mankins,  2009;  USD  (AT&L),  2011). 

The  most  convincing  evidence  of  the  TRL  scale’s  utility  may  be  found  in  its 
absence.  Consider  the  task  of  evaluating  a  new  engine  technology  for  an  aircraft 
acquisition  program  without  the  benefit  of  such  a  scale.  The  new  engine 
promises  an  appreciable  performance  advantage  over  older  technology,  but  has 
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yet  to  be  demonstrated  in  a  similar  application.  Successful  implementation  of  this 
technology  would  require  program  managers,  systems  engineers,  operational 
testers,  contract  specialists,  and  experts  in  multiple  engineering  areas  to  evolve 
specific  performance  criteria  for  the  new  engine.  Reaching  agreement  among  all 
parties  would  require  considerable  time  and  effort.  Today,  we  can  expedite  this 
process  by  identifying  the  desired  TRL.  The  common  benchmark  set  by  the  TRL 
scale  can  then  be  expanded  upon  as  necessary  within  each  discipline  involved  in 
the  acquisition  program. 

Current  challenges  in  assessing  HSI  closely  parallel  those  faced  by  NASA 
and  other  agencies  before  the  TRL  scale  was  devised.  HSI  activities  must  be 
coordinated  across  disciplines  without  the  benefit  of  a  standardized  assessment 
framework  (Phillips,  2010;  Wilson  et  al.,  2009).  Instead,  today’s  HSI  practitioners 
must  define,  prioritize,  and  translate  the  desired  state  of  human-technology 
integration  on  an  issue-by-issue  basis. 

B.  REVIEW  OF  HUMAN  FACTORS-RELATED  SYSTEM  READINESS 

MEASURES 

A  review  of  government,  commercial,  and  academic  literature  reveals  two 
relevant  models  for  assessing  HSI  during  system  acquisition.  Since  both  models 
are  new  to  acquisition,  limited  information  is  available  on  their  effectiveness  in 
actual  programs.  In  this  section,  the  merits  and  limitations  of  these  models  are 
discussed  and  compared  with  the  characteristics  of  the  TRL  scale. 

1.  System  for  Human  Factors  Readiness  Evaluation  and  Human 
Factors  Readiness  Levels 

In  2008,  the  FAA  initiated  a  sweeping  update  of  the  U.S.  air  traffic  control 
system  with  the  NextGen  program,  promising  a  top-to-bottom  overhaul  of  aging 
air  traffic  control  technologies.  The  FAA  quickly  acknowledged  the  role  of  human 
factors  in  NextGen’s  success  and  authored  a  Small  Business  Innovation  Request 
(SBIR)  seeking  industry  assistance  to  provide  specific  tools  for  this  purpose.  A 
two-phase  contract  was  awarded  to  Design  Interactive  (Dl),  Inc.,  for  a  software 
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tool  to  improve  NextGen  outcomes  by  standardizing  and  automating  human 
factors  assessment.  Dl  created  two  related  products,  System  for  Human  Factors 
Readiness  Evaluation  (SHARE),  a  computer-based  system  for  tracking  identified 
HF  issues,  and  Human  Factors  Readiness  Levels  (HFRL),  a  scale  for  prioritizing 
the  resolution  of  these  issues  (Design  Interactive,  2012;  Johnston  et  al.,  2013). 

The  first  iteration  of  SHARE  defined  nine  levels  of  HF  readiness,  aligning 
HFRL  directly  with  the  TRL  scale.  This  initial  HFRL  configuration  defined  key 
human  factors  activities,  and  then  separated  them  into  graduated  levels,  similar 
to  the  TRL  scale.  Dl’s  initial  concept  for  the  scale  is  seen  in  Figure  9. 
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Figure  9.  Initial  concept  for  HFRL  (from  Design  Interactive,  2012) 


At  Phase  I  review,  FAA  officials  identified  several  drawbacks  to  this 
approach.  Critically,  the  process-focused  levels  did  not  address  actual  outcomes 
of  HF  analysis  and  testing.  HFRL  scores  could  increase  based  on  the  completion 
of  HF  analysis  and  testing  alone,  even  if  outcomes  were  not  favorable.  This 
meant  that  even  if  a  high-priority  HF  deficiency  was  discovered  in  late-stage 
testing,  the  system’s  HFRL  rating  would  remain  unchanged.  The  FAA  concluded 
that  the  scale’s  inability  to  capture  the  importance  of  HF  issues,  regardless  of 
timing,  could  lead  to  inaccurate  ratings  (Design  Interactive,  2012). 
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Based  on  FAA  feedback,  Dl  consulted  with  government  and  commercial 
HF  practitioners  to  improve  SHARE  and  HFRL  functionality  (Design  Interactive, 
2012;  Johnston,  et  al.,  2013).  Dl  quickly  shifted  SHARE  development  toward  the 
assessment  of  actual  human-technology  performance,  allowing  for  system 
evaluation  irrespective  of  technology  state  or  acquisition  phase.  Based  on  Dl’s 
findings,  the  second  iteration  of  SHARE  was  implemented  as  an  issue-based, 
risk-guided  system  for  tracking  HF  readiness  during  system  development. 

This  second  iteration  enables  users  to  create  a  profile  for  individual  HF 
issues  as  they  arise  during  acquisition.  HF  practitioners  can  then  rate  each  issue 
using  a  standardized  FAA  risk-classification  scale,  and  catalog  the  issue  into  a 
specific  category  (e.g.,  anthropometries,  displays  and  controls).  HFRL  scores  are 
tabulated  across  categories  and  across  the  system,  based  on  highest  (worst)  risk 
scores  (Design  Interactive,  2012).  The  SHARE  software  displays  these  HFRL 
ratings  in  useful  dashboard  metrics,  allowing  FAA  users  to  track  HFRL  across 
categories  and  over  time.  An  early  SHARE  software  concept  screen  is  shown  in 
Figure  10. 
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Figure  10.  Early  SHARE  software  concept  (from  Design  Interactive, 

2012) 


SHARE  offers  a  viable  platform  for  assessing  HF-related  deficiencies,  but 
has  some  limitations.  Risk  is  well  accepted  in  federal  acquisition,  but  has 
inherent  limitations  for  capturing  the  full  spectrum  of  system  performance  (DOD, 
2006).  In  military  acquisition,  system  development  is  managed  to  ensure  that 
performance  remains  between  threshold  and  objective  values  (Defense 
Acquisitions  University,  2010).  Risk  focuses  on  issues  that  fall  below  the 
threshold  values,  and  does  not  evaluate  performance  that  is  above  this 
threshold.  This  is  useful  for  prioritizing  HF  or  HSI  deficiencies  (Harrison  & 
Forster,  2003),  but  may  be  inadequate  for  a  holistic  assessment  of  HSI 
performance. 

Nevertheless,  SHARE  and  HFRL  offer  a  robust  toolset  for  HF  issue 
identification,  classification,  and  resolution  (Design  Interactive,  2012:  Johnston, 
et  al. ,  2013).  Dl’s  shift  from  process-based  metrics  toward  performance-based 
metrics  during  software  development  suggests  a  key  lesson  for  the  development 
of  an  HSI  evaluation  framework.  SHARE  appears  well  positioned  to  standardize 
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and  improve  HF  evaluation  across  the  FAA.  For  application  to  HSI  assessment, 
the  format  and  structure  of  SHARE  may  be  useful  in  HFE  and  other  domains, 
and  warrants  careful  consideration. 

2.  Human  Readiness  Levels 

The  “Development  and  Initial  Evaluation  of  Human  Readiness  Level  (HRL) 
Framework”  (2010)  was  a  pioneering  effort  to  establish  a  meaningful  HSI  metric 
in  DOD  acquisition.  The  HRL  concept  was  conceived  by  Hector  Acosta  of  the 
USAF  711th  Human  Performance  Wing,  and  developed  into  the  HRL  framework 
by  Major  Eric  Phillips,  USAF.  Phillips  constructed  HRL  as  a  progressive  scale 
(similar  to  the  TRL)  to  synchronize  HSI  activity  and  outcomes  in  the  DOD 
acquisition  framework  (Phillips,  2010). 

Phillips  used  the  NATO  human  view  architecture  (NATO  HVA)  as  a 
conceptual  basis  for  HRL.  Introduced  by  the  UK  MoD  in  2010  to  supplement 
existing  acquisitions  procedures,  the  NATO  human  view  describes  the  primary 
areas  of  human  consideration  required  for  modern  system  design.  (For  a 
thorough  discussion  on  the  NATO  human  view,  see  the  NATO  Human  View 
Handbook,  2008.)  Using  the  NATO  human  view,  Phillips  outlined  criteria  for 
developing  a  human  readiness  level  scale  based  on  current  acquisition  practices. 
These  criteria  included  1)  adherence  to  accepted  risk  management  practices; 

2)  implementation  of  HSI  as  early  as  possible  in  the  acquisition  processs; 

3)  emphasis  on  human-centered  design  that  contributes  to  total  system 
performance;  and  4)  capturing  the  incremental  cost  of  HSI  over  the  course  of 
acquisition  (Phillips,  2010). 

To  create  HRL  levels,  Phillips  conducted  an  extensive  mapping  of  HSI 
activities,  processes,  and  products.  This  mapping  was  then  evaluated  against  the 
NATO  human  view  and  sequenced  according  to  acquisition  phase  into  nine 
levels  (Phillips  2010).  The  resulting  HRL  scale  and  descriptions  are  presented  in 
Table  1 . 
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Human  Readiness  Level 

Description 

1.  Activation  of  Human  Systems 

Integration:  base-lining  and  commitment. 

Lowest  level  of  socio-technical  readiness. 
HSI  infrastructure  is  setup  within  Systems 
Engineering  planning.  Total  system 
analysis  from  both  functional  relationship 
and  organizational  perspectives  occurs. 
Activity  examples  include  front-end 
analyses,  preliminary  functional  allocation, 
and  initial  HSI  assessment  and  plan. 

2.  Human  Systems  Integration  analysis 
suite  in  support  of  component  technology 
development. 

Significant  HSI  input  to  acquisition 
development  and  documentation  occurs. 
Activity  examples  include  initial  human- 
machine  interface  assessment,  generation 
of  Target  Audience  Description,  and 
threat/hazard  assessment. 

3.  Component  human  touch  point 
resolution  (i.e.,  human-system  interaction, 
to  include  hardware  and  software): 
refining  requirements  thresholds. 

Multiple  needs  analyses  and  studies  are 
conducted  in  support  of  requirement 
definitions.  HSI  domain  assessments 
inform  ongoing  development  actions,  as 
well.  Activity  examples  include  human-in- 
the-loop  analyses,  sub-system  hazard 
analysis,  and  HSI  plan  update  and 
revision. 

4.  Component  human  touch  point 
engineering  parameters  and  human 
performance  indicators. 

Iterative  evaluation  and  analysis  of  each 

HSI  domain  takes  place  and  provides 
critical  items  of  consideration  bearing  on 
system  design.  Activity  examples  include 
usability  testing,  development  of  human- 
centered  source  selection,  and  updating 
the  human-machine  interface  assessment. 

5.  Limited  system  human  performance 
parameters/demonstration. 

Various  HSI  assessments  and  testing  are 
performed  to  support  the  significant 
increase  in  fidelity  of  breadboard 
technology.  This  includes  supporting  “high 
fidelity”  laboratory  integration  of 
components.  Activity  examples  include 
examining  safety  and  occupational  health 
design  features  and  cognitive  task 
analyses. 

6.  Field  (relevant  environment)  validation 
of  human  performance  prototypes. 

Representative  model  or  prototype  system 
is  tested  in  a  relevant  environment. 
Evaluations  of  human  performance 
embedded  in  demonstration  system  occur 
and  HSI  predictive  models  are  updated. 
Activity  examples  include  testing  human 
reliability  and  usability  of  prototype  in  a 
high-fidelity  laboratory  environment  or  in 
simulated  operational  environment. 
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7.  Final  Developmental  Test  & 
Evaluation/human  performance 
parameters. 

Significant  HSI  participation  in  system  test 
events  occurs.  Iterative  evaluation  and 
analysis  of  each  HSI  domain  continues  as 
well.  Activity  examples  include  error  and 
fault  analysis  to  cover  human  error 
performance,  equipment  operability,  safety 
procedures,  and  error  recovery 
mechanisms. 

8.  Operational  Test  &  Evaluation/human 
performance  parameters. 

Special  human-centric  analyses  are 
conducted  to  update  thresholds, 
objectives,  and  evolving  criteria  for 
Operational  Test  &  Evaluation.  Iterative 
evaluation  and  analysis  of  each  HSI 
domain  continues  as  well.  Activity 
examples  include  system  hazard  analysis 
and  HSI  domain  tradeoff 
studies. 

9.  Sustainment:  Initiation  of  capability  gap 
feedback  cycle. 

Extensive  and  iterative  review  and 
verification  of  fielded  system  begins,  as 
well  as  post-product  improvement 
evaluations  for  the  next  incremental  builds. 
Activity  examples  include  post-fielding 
training  evaluation  analysis  and  sustaining 
a  hazard  analysis  for  the  fielded  system. 

Table  1.  HRL  nine-level  scale  and  definitions 
(from  Phillips,  2010) 


HRL  levels  share  essential  features  with  the  first  HFRL  concept  proposed 
by  Design  Interactive.  Both  frameworks  were  constructed  to  coincide  with  the 
TRL  scale  and  both  focus  on  the  completion  of  acquisition  activities.  HRL  is 
hindered  by  many  of  the  same  drawbacks  identified  by  the  FAA  in  their 
evaluation  of  Dl’s  initial  work.  Like  HFRL,  HRLs  take  a  process-oriented 
approach.  The  HRL  scale  provides  for  standardization  and  increased  awareness, 
but  does  not  assess  or  account  for  HSI  efficacy.  This  can  lead  to  inaccurate 
ratings.  Consider,  for  example,  the  detailed  definition  for  HRL  Level  5  (Table  1): 

Various  HSI  assessments  and  testing  are  performed  to  support  the 
significant  increase  in  fidelity  of  breadboard  technology.  This 
includes  supporting  high  fidelity  laboratory  integration  of 
components.  Activity  examples  include  examining  safety  and 
occupational  health  design  features  and  cognitive  task  analyses. 
(Phillips  2010) 
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Even  if  all  activities  in  this  HRL  level  are  completed,  the  extent  to  which 
human-technology  integration  has  contributed  to  system  performance  remains 
unclear.  Many  questions  remain  unanswered,  including: 

•  Were  the  results  of  HSI  assessments  favorable? 

•  Did  the  state  of  human-technology  integration  improve  or  degrade 
overall  system  performance? 

•  What  did  testing,  evaluation,  and  analysis  reveal  about  the  human 
contribution  to  system  goals? 

These  questions  reveal  an  important  difference  between  technology  readiness 
measures  (i.e.,  TRL)  and  potential  human-technology  integration  measures. 
Technology  outcomes  are  more  compatible  with  the  progressive  trajectory 
provided  by  the  TRL  scale.  Reliability  analysis,  for  example,  examines  system 
performance  over  time  based  on  individual  technology  components  either  being 
operable  or  not;  mechanical  components  either  perform  or  fail;  software  code 
segments  function  or  crash,  and  tend  to  do  so  in  predictable  ways  (Blanchard  & 
Fabrycky  2011).  When  applied  to  humans,  however,  this  approach  can  become 
problematic  (Sanders  &  McCormick  1993).  Subjective  assessments  are  typically 
required  to  evaluate  human  performance,  and  human  behavior  can  vary 
considerably  over  time.  As  will  be  discussed  in  the  following  chapters,  assessing 
how  humans  interact  with  technology  requires  a  change  in  the  way  we  think 
about  humans  and  technology  components  in  complex  systems. 

C.  POTENTIAL  SUCCESS  FACTORS  FOR  THE  HSI  FRAMEWORK 

The  TRL  scale’s  many  attractive  qualities  have  made  it  the  “coin  of  the 
realm”  for  systems  assessment  (Mankins,  2009).  As  suggested  by  the  limitations 
identified  with  HRL  and  SHARE,  however,  a  different  vehicle  is  required  for 
assessing  human-technology  integration.  The  following  attributes  are  suggested 
as  criteria  for  a  meaningful  HSI  assessment  framework: 
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1.  Simple  and  Discipline  Independent 

The  HSI  framework  must  be  easily  understood.  Simple,  discipline- 
independent  language  will  increase  the  framework’s  acceptance,  implementation, 
and  application  across  disciplines. 

2.  Surpassing  Risk-Based  Measures 

Measuring  the  contribution  of  HSI  must  go  beyond  risk-based  approaches 
to  encompass  all  effects  on  total  system  performance.  Considering  risk  alone 
may  not  offer  the  breadth  required  for  a  complete  assessment  of  HSI 
performance. 

3.  Performance  Focused 

The  HSI  framework  must  focus  on  evaluating  the  adequacy  of  the  human- 
technology  integration,  measured  in  terms  of  total  system  performance.  This 
criterion  is  a  departure  from  previous  approaches,  which  focus  largely  on  process 
completion  rather  than  performance  outcomes. 

By  incorporating  the  helpful  qualities  of  the  TRL  scale  and  building  upon 
the  useful  aspects  of  HRL  and  SHARE,  a  more  comprehensive  framework  for 
human-technology  integration  can  be  envisioned.  The  following  chapters  explore 
these  concepts  and  offer  a  framework  for  assessing  human-technology 
integration. 
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III.  CONCEPTUALIZING  HSI  ACTIVITY  DURING  SYSTEM 

ACQUISITION 


A.  MODEL  FOUNDATIONS 

This  chapter  develops  a  conceptual  model  for  HSI  in  system  acquisition. 
The  model  is  based  on  three  fundamental  tenets:  humans  and  technology  are 
the  primary  actors  in  the  system;  accomplishing  system  objectives  is  the  raison 
d’etre  for  the  interaction  between  humans  and  technology;  and,  humans  and 
technology  have  different  patterns  of  emergence  during  system  acquisition. 

1.  Human  and  Technology  Roles  in  Modern  Military  Systems 

The  allocation  of  functions  to  either  humans  or  machines  is  one  of  the 
most  important  design  decisions  in  HF  and  HSI.  As  early  as  the  1950s, 
researchers  suggested  rules  of  thumb  for  assigning  system  tasks  to  the  most 
capable  party.  The  “Fitts  list”  (Figure  11),  developed  by  the  industrial 
psychologist  Paul  Fitts,  is  perhaps  the  best  known  HF  design  guideline  (de 
Winter  &  DODou,  2011). 
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Figure  1 1 .  Man  and  machine  capabilities  circa  1950 
(from  de  Winter  &  DODou,  2011) 


As  technological  capability  and  the  understanding  of  human  cognitive 
processes  have  improved,  the  relative  strengths  and  weaknesses  of  man  and 
technology  have  been  the  subject  of  continual  debate  (de  Winter  &  DODou, 
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2011;  Dekker  &  Woods,  2002).  In  this  thesis,  the  respective  merits  of  humans 
and  technology  and  their  ideal  confluence  are  set  aside.  What  we  gather  from 
more  than  five  decades  of  discourse  is  that  the  primary  actors  in  the  system  are 
either  people  or  technology  (Blanchard  &  Fabrycky,  2011).  Following  this  logic, 
functions  that  cannot  be  performed  by  one  party  must  be  carried  out  by  the  other 
as  they  work  in  tandem  to  accomplish  system  goals. 

2.  Combining  Humans  and  Technology  to  Accomplish  System 

Goals 

In  modern  systems,  the  humans,  technological  tools,  and  system  goals 
are  inextricably  linked  (Kuuti,  1996;  Shattuck  &  Miller,  2006).  This  relationship  is 
the  focus  of  activity  theory  (AT),  originating  from  Soviet  psychology  in  the  1970s. 
Activity  theory  asserts  that  to  develop  a  meaningful  understanding  of  human- 
technological  activity,  we  must  examine  three  fundamental  aspects  of  interaction 
(Kuuti,  1996): 

•  Subject:  the  user  or  user  group,  including  operators, 
maintainers,  and  supporters 

•  Tools:  technological  elements  and  artifacts  used  to  accomplish 
system  tasks 

•  Object:  the  objective  or  goal  of  system  activity 

By  considering  the  interaction  of  all  three  elements,  AT  provides  a  basic 
framework  for  analysis  (for  a  more  nuanced  discussion  of  AT,  see  Kuuti,  1996  or 
Kaptelinin  &  Nardi,  1996). 

AT  is  a  foundational  concept  in  user  experience  design  and  human- 

computer  interaction,  where  it  has  informed  the  merging  of  humans  and 

technology  for  more  than  two  decades  (Kuuti,  1996).  We  propose  that  AT  offers 

similar  utility  within  military  acquisition.  In  system  acquisition,  requirements 

derived  from  broad  system  objectives  guide  system  development,  shaping  the 

ways  in  which  technology  and  users  are  integrated.  This  maps  well  to  the 

theoretical  elements  proposed  in  AT:  the  “subject”  is  analogous  to  the  system 

users,  including  operators  maintainers,  and  supporters;  “tools”  are  analogous  to 
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the  elements  operated  by  users;  and  the  “object”  is  the  performance  objectives  of 
the  total  system.  Figure  12  displays  this  relationship. 


Technology 


System  Performance 

Figure  12.  Activity  theory  (AT)  in  military  acquisition  (after  Kuuti,  1996) 

The  transformation  in  Figure  12  depicts  the  interaction  of  subject  and  tool 
within  the  context  of  the  object  (the  performance  objective)  to  produce  an 
outcome.  In  the  acquisition  of  a  military  system,  this  transformation  occurs  as 
users  interact  with  system  technology  to  perform  functions.  This  process 
captures  the  essence  of  HSI.  Thus  Figure  12  depicts  the  total  system 
performance  that  results  from  human-technology  interaction. 

B.  A  MODEL  FOR  HSI  IN  ACTION 

Bringing  together  the  concepts  above,  a  consolidated  view  of  HSI  in 
military  systems  acquisition  is  derived.  HSI  is  represented  as  the  merging  of 
human  and  non-human  elements,  unified  by  system  objectives. 

1.  The  Proposed  HSI  Activity  Model 

The  foundational  concepts  presented  in  this  research  can  be  unified  into  a 
conceptual  model.  As  depicted  in  Figure  13,  the  human  and  technological 
elements  of  a  system  derive  from  separate  origins  and  merge  to  accomplish 


25 


system  objectives.  The  accomplishment  of  these  objectives  is  measured  by  total 
system  performance  goals  (at  right). 


Figure  13.  Proposed  HSI  Activity  Model 


The  system  evolves  throughout  the  SE  process,  as  suggested  by  the 
movement  from  left  to  right  in  Figure  13.  Technology  matures  over  time  as  an 
acquisition  proceeds.  Humans  “come  as  they  are,”  with  preexisting  capabilities 
and  limitations.  What  technology  permits  (affordances)  and  what  it  does  not  allow 
(constraints),  must  align  with  both  what  a  human  can  do  (capabilities)  and  cannot 
do  (limitations).  Human  and  technological  elements  interact  as  the  system  takes 
shape,  represented  by  the  convergence  of  arrows  in  the  model.  Orange,  green, 
and  blue  boundaries  enclose  the  respective  system  assessment  areas. 

2.  Human  and  Technological  Contributions  to  the  System 

As  defined  by  Norman,  affordances  are  those  fundamental  characteristics 
that  determine  how  a  “thing”  can  possibly  be  used  (Norman,  1988).  We  use  the 

26 


term  technology  affordances  to  capture  the  ways  in  which  technological  elements 
can  be  used  to  perform  system  functions.  For  example,  steering  wheels  afford 
turning,  which  creates  mechanical  leverage  and  controls  movement.  However, 
technologies  also  have  needs  relative  to  a  system.  The  steering  process  requires 
human  perception,  strength,  judgment,  and  other  capabilities  to  perform  its 
function.  We  define  these  external  requirements  as  technology  constraints. 
These  affordances  and  constraints  are  revealed  as  a  technology  is  introduced 
into  system  to  supplement  human  capabilities  and  overcome  limitations. 

Like  technology,  humans  offer  capabilities  that  can  be  used  in  system 
tasks  such  as  physical  strength,  vision,  audition,  memory,  and  motor  skills. 
Humans  also  have  needs  relative  to  a  system,  such  as  a  breathable  atmosphere, 
lighting,  and  protection  from  mechanical  shock,  vibration,  and  excessive 
gravitational  forces  (NASA,  2010).  Man  and  machine  alike  have  something  they 
offer  and  something  they  need.  Table  2  provides  examples  of  this 
interdependency  from  the  human  side,  at  different  levels  of  task  complexity. 


Example  Human  Capabilities 

Example  Human  Needs 

Human 

Function 

Level 

Activity 
Theory  Level 

Situation  analysis;  complex 
decision  making;  sense-making; 
option  generation 

Decision  support,  contextualized 
sensor  information;  process  control 

Job 

Operation 

Activity  Level 

Situation  awareness,  decision 
making  automation  oversight, 
meta-system  monitoring 

Control  feedback;  temporal 
indicators,  system  status,  spatial 
orientation,  mode  awareness 

Duty 

Action  Level 

Subsystem  monitoring,  rule 
interpretation,  system  calibration 

Sensory  input  (haptic,  visual, 
auditory  etc.);  physical 
functionality,  menu  control 

Task 

Visual  scanning,  target  association, 
parameter  monitoring 

Physical  access;  physical 
accommodation;  visual  access 

Subtask 

Operation  Level 

Table  2.  Human  capabilities  and  limitations 
(after  Shattuck  &  Miller,  2006;  Blanchard  &  Fabrycky,  2011;  Nardi 

&  Kutti,  1996) 
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3.  Different  Paths  of  Emergence  during  System  Acquisition 

Human  performance  is  essential  to  modern  military  systems  (Booher, 
2003).  Like  measures  of  propulsion,  mechanical  strength,  and  computing  power, 
measures  of  human  physiological,  sensory  and  cognitive  characteristics  are 
fundamental  considerations  for  effective  design  (Miller  et.  al. ,  2003).  The 
development  of  the  FIM-92  “Stinger”  man-portable  air  defense  system  offers  a 
superb  example.  In  the  final  stages  of  testing,  the  fully  assembled  system 
(sensors,  targeting  system,  launcher,  missile,  etc.)  was  reported  to  have  a  kill 
probability  of  60  percent.  With  a  human  operator  included  in  the  system, 
however,  the  actual  probability  of  kill  fell  to  30  percent  (Booher,  2003).  Poor 
human-technology  interfaces  led  to  a  significant  decrement  in  performance.  The 
lesson  for  system  design  is  clear:  humans  and  technology  must  both  be 
considered  as  contributors  to  system  performance  (Booher,  2003). 

The  FIM-92  example  underscores  a  critical  fact  about  humans  and 
technology.  Technology  and  technology  capacity  can  be  evolved  relatively 
quickly  over  a  period  of  months  or  years  to  meet  system  needs.  Human 
capabilities,  however,  evolve  at  a  much  slower  rate  over  millennia. 

Consider  the  human  attributes  relevant  to  a  Coast  Guard  coxswain  piloting 
a  small  boat.  The  coxswain’s  muscular  strength  and  anthropometric 
measurements  are  essential  considerations  for  the  tasks  at  hand  such  as 
steering,  throttle  adjustment,  and  control  actuation.  These  physical  attributes  can 
be  measured  directly  and  objectively,  and  their  underlying  mechanisms  are 
relatively  well  understood  (NASA,  2010;  Pheasant  &  Haslegrave,  2005). 
Alternatively,  consider  the  attention  capacity  and  situational  awareness  of  the 
same  coxswain.  These  attributes  may  be  measured  only  indirectly  (Miller, 
Crowson  &  Narkevicius,  2003;  NASA  2010)  and  their  underlying  cognitive 
mechanisms  are  the  subject  of  ongoing  scientific  study  (Shattuck  &  Miller,  2006). 

By  virtue  of  these  diverse  capabilities,  HSI  must  encompass  a  wide  variety 
of  measures,  each  with  computational  limitations.  HSI  measures  range  from 
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nominal  and  ordinal,  which  have  certain  computational  restrictions,  to  interval 
and  ratio  measures  that  allow  added  statistical  precision  (Stangor,  2010). 
Examples  of  different  types  of  measures  relevant  to  HSI  are  illustrated  in  Table  3. 


Scale 

Definition 

HSI/Acquisition  Example 

Nominal 

A  categorical  variable;  offers 
categorization  but  cannot  be  ranked 

Gender,  ethnicity,  operational  specialty  code, 
decision  making  variables,  etc. 

Ordinal 

A  qualitative  variable  with  values  that 
can  be  rank  ordered;  distance  between 
values  is  not  specified 

Operator  usability  rating,  perceived  level  of 
difficulty,  perceived  operator  fatigue,  situation 
awareness  score,  color  coding,  etc. 

Interval 

Variable  where  differences  in  values 
reflect  distance  between  the  values; 
zero  point  is  arbitrary 

Temperature,  seating  orientation  (degrees), 
direction  of  airflow,  operator  intelligence 
quotient,  etc. 

Ratio 

Variable  with  defined  ratio  values  and 
a  true  zero  point 

Pounds  of  force  required,  button  relief, 
processing  speed,  luminance,  etc. 

Table  3.  Measurement  theory  for  military  HSI 
(after  Stangor,  2010) 


Although  the  capabilities  and  limitations  of  individual  persons  vary  widely, 
they  do  so  within  a  relatively  consistent  range.  Intelligence  quotient,  visual  acuity, 
or  grip  strength,  for  example,  vary  widely  among  individuals  yet  fall  within  a 
predictable  range  across  populations  (Chapanis,  1996;  Miller  et.  al. ,  2003). 
Technology,  by  contrast,  can  be  constructed  from  the  ground  up  with  attributes 
chosen  to  fulfill  a  specific  purpose.  The  materials,  types  of  technology,  and 
underlying  architecture  of  a  system  can  be  re-composed  as  needed  (Blanchard  & 
Fabrycky,  2011).  Humans  are  far  more  constrained  as  a  system  variable.  Human 
performance  variability  can  be  reduced  to  some  extent  through  selection, 
training,  and  manipulation  of  factors  affecting  human  states.  Unlike  technology, 
however,  recomposing  or  re-engineering  human  capabilities  is  simply  not  a 
viable  system  design  option. 

Humans  and  technology  develop  from  different  origins,  and  contribute  to 
system  functions  in  very  different  ways.  For  this  reason,  our  model  of  HSI  Activity 
incorporates  separate  pathways  for  Humans  and  Technology.  Put  simply, 
technology  may  or  may  not  be  ready  for  the  systems  we  desire,  but  humans  are 
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ready  now.  It  is  the  responsibility  of  those  who  design  the  system  to  do  so  in  a 
manner  that  provides  the  affordances  and  constraints  that  align  with  the 
capabilities  and  limitations  of  the  humans  who  will  operate,  maintain  and  support 
the  system. 

4.  Advantages  of  the  Proposed  HSI  Activity  Model 

The  proposed  HSI  activity  model  adds  to  existing  HSI  and  HF  models  by 
portraying  the  core  activity  of  HSI  as  a  single  figure  (for  more  on  founding  HSI 
models,  see  Booher,  2003).  This  feature  is  essential  for  improving  HSI 
understanding  and  awareness,  as  advocated  in  earlier  chapters.  The  HSI  activity 
model  lays  the  groundwork  for  the  HSI  assessment  framework  proposed  in 
Chapter  IV. 

The  HSI  activity  model  doubles  as  a  design  paradigm.  Adopting  the  view 
proposed  in  the  model,  technology  affordances  are  considered  in  concert  with 
their  constraints,  and  human  capabilities  are  considered  in  concert  with  the 
limitations  they  bring.  This  provides  valuable  perspective  for  design  before  the 
integration  of  user  and  technology  is  attempted. 
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1.  Change  in  Projected  2.  New  Target 

Operators  (E-6  to  E-3)  Audience  Description 


1.  New  Technology 
Introduced 


2.  New  Technology 
Functionality 


Figure  14.  HSI  Activity  Model  example  application 


Figure  14  illustrates  an  application  of  the  HSI  activity  model.  Human 
capabilities  and  limitations  are  integrated  progressively  with  technology 
affordances  and  constraints.  First,  human  capabilities  and  limitations  emerge, 
based  on  the  selection  of  users  (top  left).  As  the  human  contribution  emerges, 
technology  is  introduced,  offering  new  functionality  but  also  creating  new 
demands  on  users  (bottom  center).  The  contributions  and  needs  of  both  are 
integrated  to  achieve  system  performance  goals  (right). 

5.  Limitations  of  the  HSI  Activity  Model 

Military  acquisition  often  includes  the  iterative  introduction  of  new 
technologies  over  multiple  years  of  system  development.  In  addition,  user  groups 
(operators,  maintainers,  and  supporters)  may  be  recomposed  to  meet  changing 
program  needs.  It  is  conceded  that  the  updating  of  human  and  technological 
needs  will  probably  not  occur  in  the  progressive,  neatly  discernable  fashion 
depicted  by  the  alternating  gray  arrows  in  Figure  14. 
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Like  other  HF  models,  the  proposed  HSI  activity  model  is  an  abstraction  of 
a  complex  and  multifaceted  process;  the  model  has  the  virtue  of  simplifying  and 
generalizing  HSI  activities  so  they  can  be  readily  understood  (Norman,  1988; 
Parasuraman,  Sheridan,  &  Wickens,  2000).  The  inherent  tradeoff  with  this 
abstraction  is  that  precise  measures  have  limited  applicability.  This 
oversimplification  is  made  with  a  deliberate  purpose:  increasing  HSI 
understanding  and  awareness.  The  equally  important  task  of  measuring  HSI  will 
be  explored  in  the  next  chapter. 
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IV.  DEVELOPMENT  OF  A  FRAMEWORK  FOR  ASSESSING  HSI 
DURING  SYSTEM  ACQUISITION 


This  chapter  documents  the  creation  of  a  framework  for  assessing  HSI  in 
system  acquisition.  The  framework  was  created  in  consultation  with  NPS  HSI 
faculty  and  refined  with  HSI  practitioners  from  the  Coast  Guard  Human  Systems 
Integration  Division  (CG-1B3)  during  a  series  of  workshops.  As  a  culminating 
exercise,  practitioners  applied  the  framework  to  assess  a  current  acquisition 
program,  the  U.S.  Coast  Guard  Fast  Response  Cutter,  during  a  mock  design 
review.  The  resulting  Comprehensive  Human  Integration  Evaluation  Framework 
(CHIEF)  elements  are  presented  at  the  conclusion  of  the  chapter. 

A.  STEP  ONE:  DEVELOPING  AN  HSI  FIGURE  OF  MERIT 

A  “figure  of  merit”  (FOM)  is  used  in  engineering  to  develop  a  concise 
expression  of  the  "degree  of  goodness"  among  competing  alternatives  during 
design  (Lee,  2011).  Developing  a  FOM  allows  engineers  to  establish  a  value 
hierarchy  and  evaluate  design  alternatives  based  on  their  relative  merit  in  a 
particular  design  application.  According  to  veteran  employees  at  NASA,  the  TRL 
scale  was  constructed  specifically  as  a  FOM  (Mankins,  2009).  Early  in  this 
project,  we  decided  to  develop  a  full  FOM  for  HSI. 

First,  a  value  hierarchy  was  created  for  each  HSI  domain.  The  objective  of 
this  effort  was  to  create  a  generalized,  system-independent  definition  of  HSI 
success  and  failure  for  each  HSI  domain.  As  a  benchmark  for  success,  HSI  FOM 
language  was  developed  to  match  the  concise  language  employed  by  the 
NASA/DOD  TRL  scale.  Statements  describing  the  desired  state  and  undesired 
state  of  each  HSI  domain  were  crafted.  Concepts  and  language  were 
synthesized  from  the  Coast  Guard  Major  Systems  Acquisition  Manual  (MSAM), 
the  Defense  Acquisition  Guidebook  (DAG),  the  U.S.  Air  Force  Human  Systems 
Integration  Handbook,  the  U.S.  Army  MANPRINT  Handbook,  and  academic 
textbooks. 
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Developing  TRL-like  definitions  for  each  HSI  domain  required  a  significant 
reduction  in  verbiage.  This  was  achieved  by  removing  overly  specific  phrases, 
context,  and  design  instances  from  main  definitions,  and  describing  them 
separately  as  design  considerations.  Table  4  shows  the  resulting  HSI  FOM  for 
the  domains  of  personnel,  manpower  and  PS&T. 


Domain 

Manpower 

Personnel 

Performance  Support  & 
Training 

The  correct  number  and  mix 

Human  performance  factors 

Competency  requirements  are 

of  personnel  required  for 

are  identified  and  matched  to 

established  to  reflect  end- 

<D 

system  operation, 

system  tasks  and  workload; 

user  tasks  and  workload; 

4— • 

CO 

maintenance  and  support  is 

criteria  are  developed  to 

knowledge,  skill  and  ability 

"O 

a> 

identified;  manning  reflects 

effectively  recruit,  select  and 

gaps  are  appropriately 

’cn 

the  full  range  of  operations, 

train  personnel  for  safe, 

identified  and  training  system 

o 

work  demands,  operating 

efficient  and  effective  system 

is  developed  to  match  end- 

conditions  and  deployment 
scheme  for  the  system. 

operation. 

user  learning  needs. 

manpower  estimates, 

Target  audience  descriptors, 

Knowledge,  skills,  aptitudes, 

individual  competencies,  team 

knowledge,  skills,  aptitudes, 

experiences,  cognitive 

to 

competencies,  occupational 

experiences,  traits,  cognitive 

abilities,  cognitive  style, 

C 

specialties,  resilience,  task 

abilities,  cognitive  style, 

personality  factors; 

(J 

c 

capacity,  net  workload,  watch 

personality  factors;  workload, 

competencies,  task  demands, 

o 

u 

schedules,  operating 

cognitive  demands,  task 

cognitive  demands,  subject 

c 

.tap 

conditions,  ship  fill, 

reguirements,  task  duration, 

matter,  audience,  learning 

<D 

Q 

deployment  schedules, 

taskfreguency,  task  design, 

styles,  delivery  mode,  training 

chronic  fatigue  levels 

task  environment,  operating 
conditions 

infrastructure,  instructional 
technology,  instructional 
design,  evaluation  metrics 

The  mix  and/or  number  of 

Individual  performance  factors 

Competency  requirements  are 

personnel  specified  for  the 

are  poorly  matched  to  system 

not  matched  with  end-user 

system  exceeds  allowable 

tasks,  workload  and  skill 

tasks  and  workload; 

CD 

-*-> 

manning  levels;  manning  is 

requirements.  Criteria  are  not 

knowledge  and  skills  gaps  are 

ru 

■4— • 

co 

insufficient  for  the  full  range 

adequately  established  or 

inadequately  understood  or 

"D 

CU 

of  operations,  work  demands, 

implemented  for  recruiting, 

defined;  training  system  is 

i_ 

'to 

operating  conditions  and 

selecting,  training  and 

incompatible  with  end-users 

"D 

C 

deployment  scheme  of  the 

retaining  personnel  that  meet 

learning  needs  and 

3 

system. 

or  exceed  the  required 
aptitudes  and  traits  for  the 
system. 

environment;  delivery  and 
retention  of  job-relevant 
knowledge,  skills  and  abilities 
is  inadequate. 

to 

<D 

DAG  Chapter  6.3 

DAG  Chapter  6.3 

DAG  Chapter  6.3 

c 

O) 

AF  HSI Handbook 

AF  HSI  Handbook 

AF  HSI  Handbook 

1— 

CD 

M— 

MANPRINT  Handbook 

MANPRINT  Handbook 

MANPRINT  Handbook 

CD 

CC 

CG  MSAM 

CG  MSAM 

CG  MSAM 

Table  4.  Figure  of  merit  for  three  HSI  domains 
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The  full  HSI  FOM  including  statements  for  all  Coast  Guard  HSI  domains, 
is  included  in  Appendix  A.  As  seen  in  the  appendices,  outcomes  in  each  HSI 
domain  were  successfully  categorized  using  the  FOM  approach.  This  process 
revealed  that  a  key  element  of  NASA  TRL  could  be  applied  to  assessing  HSI: 
that  is,  succinct,  generalizable  desired  states  could  be  articulated  in  a  way  similar 
to  TRL  definitions.  The  simple,  discipline-independent  language  desired  for  the 
HSI  Framework  was  now  available. 

1.  Centering  the  Scale:  Balancing  Technology  and  Human  Needs 
for  Total  System  Performance 

Developing  the  HSI  FOM  required  identification  of  desired  and  undesired 
states  in  each  HSI  domain.  As  a  result  of  this  process,  the  central  theme 
presented  in  the  HSI  Activity  Model  was  confirmed:  optimal  performance  in  each 
HSI  domain  was  best  described  as  the  balancing  of  needs  between  humans  and 
technology  for  the  accomplishment  of  system  goals.  This  was  an  important 
confirmation  of  the  activity-based  view  of  HSI  presented  in  the  last  chapter. 

The  manpower  domain  offered  an  interesting  case  study.  Manpower 
activities  in  acquisition  typically  focus  on  crafting  the  correct  mixture  of  personnel 
and  positions  to  meet  system  performance  needs.  This  typically  entails  matching 
the  competencies  and  capacity  of  personnel  to  the  work  presented  by  the  system 
(Defense  Acquisitions  University,  2010).  Developing  a  FOM  for  this  activity 
involved  a  critical  question:  For  a  particular  system,  what  characteristics  make  a 
given  manpower  solution  more  or  less  desirable  than  another? 

In  an  era  of  diminishing  budgets,  cost  typically  anchors  our  assessment  of 
acquisition  alternatives.  The  preferred  solution  offers  the  desired  system 
performance  attributes  at  a  given  cost  level.  Thus,  establishing  a  preferred 
manpower  alternative  requires  that  we  establish  the  candidate  that  will  lead  to 
the  best  possible  total  system  performance  (Barber,  2011;  Defense  Acquisitions 
University,  2010).  By  examining  the  dynamic  interplay  between  the  capabilities 


35 


and  limitations  of  humans  and  the  affordances  and  constraints  of  technology,  we 
can  identify  the  most  viable  manpower  alternatives. 

2.  Assembling  HSI  Understanding:  From  Detailed  Standards  to 
Management  Indicators 

Measuring  HSI  efficacy  is  a  particularly  challenging  endeavor.  The  single 
domain  of  HFE  alone  requires  diverse  policies,  standards  and  design  criteria. 
The  Department  of  Defense  Design  Criteria  Standard-Human  Engineering  (MIL- 
STD-1472G)  is  evidence  of  this  fact,  with  detailed  standards  governing  every 
aspect  of  human-technology  interaction  across  the  full  spectrum  of  military 
systems  and  environments.  NASA’s  Human  Integration  Design  Handbook 
(NASA/SP-20 10-3407)  offers  similarly  comprehensive  guidance,  with  more  than 
1 ,000  pages  of  detailed  standards. 

Despite  their  extraordinary  level  of  detail  and  breadth,  interpreting  and 
applying  these  standards  can  be  challenging  (Chapanis,  1996;  Lockett  &  Powers, 
2003).  Consider  the  example  of  designing  a  single  button  for  a  control  console. 
More  than  ten  design  parameters  for  the  configuration  of  the  button  can  be  found 
in  MIL-STD-1472G.  Example  parameters  include  button  relief  (depth  of  press), 
spacing,  size,  shape,  resistance  (force  required  for  actuation),  arrangement  in 
relation  to  function,  labeling,  luminance  (ambient  light),  lighting  (button 
illumination),  reflectivity  and  guarding.  Figure  15  offers  a  sampling  of  these 
standards. 
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Dimensions  <D.  Diameter) 

Resistance 

Fingertip 

Thumb 

Palm 

Bare 

hand 

Gloved 
hand  - 

Bare 

hand 

Gloved 
hand  - 

Bare 

hand 

Gloved 
hand  - 

Single 

finger 

Different 
fingers  - 

Thumb/pal 

m 

Minimum 

10  mm 
(0.4  m) 

19  mm 
(0.75  in) 

19  mm 
(0.75  in) 

25  mm 
(1.0  in) 

40  mm 
(1.6  in) 

50  mm 
(2.0  in) 

2.8  N 
(10  oz) 

1.4  N 
(5  oz) 

2.8  N 
(10  oz) 

Maximum 

25  mm 
(1.0  m) 

25  mm 
(1.0  in) 

- 

70  mm 
(2.8  in) 

“ 

11.0  N 
(40  oz) 

5.6  N 
(20  oz) 

23.0  N 
(80  oz) 

Displacement  (A) 

Fingertip 

Thumb  or  palm 

Minimum 

2.0  mm  (0.08  m) 

3-0  mm  (0.12  in) 

Maximum 

6.0  mm  (0.25  in) 

38  mm  (1.5  m) 

Separation  (S) 

Single  finger 

Single  finger  sequential  - 

Different  finger  - 

Thumb  or  palm  - 

Bare 

Gloved 

Minimum 

13  mm  (0.5  in) 

25  mm  (1.0  in) 

6.0  mm  (0.25  in) 

6.0  mm  (0.25  in) 

25  mm  (1.0  m) 

Preferred 

50  mm  (2.0  in) 

- 

13  mm  (0.5  in) 

13  mm  (0.5  in) 

150  mm  (6.0  in) 

Figure  15.  Button  configuration  parameters  (from  MILSTD  1472-G) 


Detailed  HSI  and  HFE  standards  offer  essential  guidelines  for  human- 
technology  interface  design.  For  their  effective  application,  however,  additional 
factors  must  be  taken  into  account.  In  practice,  system  performance  is  shaped  by 
user  characteristics  and  the  circumstances  of  the  task  (Chapanis,  1996;  Kuuti, 
1996).  To  evaluate  the  efficacy  of  a  human-technology  interface  design,  we  must 
extend  our  analysis  to  include  other  aspects  of  the  task  activity. 

Our  discussion  in  Chapter  III  advocated  an  activity-based  evaluation  of 
HSI.  To  carry  this  out,  we  require  specific  measures:  HSI  tools,  techniques, 
approaches,  and  methods  (TTAMs)  that  capture  the  interaction  of  configured 
system  technology  with  representative  user  characteristics,  in  a  relevant  work 
context.  For  brevity,  we  refer  to  these  as  aggregating  measures. 
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A  survey  of  HSI  literature  reveals  a  host  of  TTAMs  suitable  for  use  as 
aggregating  measures  (UK  MoD,  2008;  FAA,  2014).  Dynamic  anthropometric 
modeling  (DAM)  is  an  example  within  the  HFE  domain,  as  seen  in  Figure  6  of 
Chapter  I.  Simulated  human  users  and  configured  technology  are  placed  into  a 
virtual  prototype  through  computer  modeling;  the  practitioner  can  then  evaluate 
technology  features,  such  as  buttons  or  control  configuration,  in  concert  with 
relevant  user  characteristics,  such  as  anthropometric  measures  or  range  of 
motion  (Pheasant  &  Flaslegrave,  2005).  Practitioners  can  then  assess  human- 
technology  interaction  outcomes  for  the  system. 

Virtual  or  physical  prototyping  provides  a  convenient  example  of  an 
aggregating  measure  in  HFE;  however,  suitable  measures  in  other  domains  are 
also  available.  The  fatigue  avoidance  scheduling  tool  (FAST),  for  instance, 
serves  as  an  aggregating  measure  for  the  manpower  domain.  FAST  uses  the 
72-hour  sleep  history  of  individuals  to  predict  their  alertness  in  tasks  that  require 
vigilance  (Eddy  &  Hursh,  2006).  Manning  levels  can  then  be  adjusted  to  ensure 
that  the  individuals  performing  the  task  are  capable  of  the  vigilance  required. 

Once  sufficient  evidence  of  HSI  outcomes  is  established  for  the  selected 
HSI  domain,  the  impact  of  human-technology  interaction  on  total  system 
performance  can  be  assessed.  Figure  16  illustrates  an  example  of  this 
measurement  process  for  the  HFE  domain.  HFE  policies  and  standards  form  the 
basis  for  engineer  and  practitioner  activity;  system  attributes  are  expected  to  be 
developed  based  on  these  standards.  The  users  are  evaluated  as  they  interact 
with  the  system  in  the  work  context,  using  aggregating  measures.  These 
measures  are  then  combined  to  formulate  an  assessment  of  the  HFE  domain  in 
question,  Evaluations  of  the  other  HSI  domains  are  combined  into  a 
comprehensive  HSI  assessment,  using  the  framework  proposed  later  in  this 
chapter. 
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Figure  16.  The  “Iceberg”:  From  HFE  standards  to  HSI  performance 


Our  study  of  technology  readiness  assessment  in  Chapter  II  revealed  a 
key  lesson:  system  assessment  measures  must  be  suitable  for  a  broad 
audience.  We  propose  that  this  is  equally  true  for  assessing  HSI.  Acquisition 
program  personnel  often  have  limited  exposure  to  the  HSI  discipline.  Despite 
this,  modern  acquisition  requires  program  managers,  systems  engineers, 
contracting  specialists  and  others  to  acquire  and  apply  some  basic  HSI 
knowledge.  The  proposed  HSI  evaluation  framework  aims  to  augment  acquisition 
practitioner  knowledge  by  offering  a  consolidated  assessment  of  HSI  that  is 
rooted  in  quantitative  data  and  empirically  derived  standards. 

B.  STEP  TWO:  DEVELOPING  A  HUMAN  INTEGRATION  EVALUATION 
FRAMEWORK 

With  the  HSI  FOM  and  measurement  approach  established,  attention 

shifted  to  development  of  an  assessment  format.  Assembling  HSI  information 

across  multiple  domains  presented  a  challenge.  A  familiar  HFE  assessment  tool 
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provided  an  analog  to  meet  this  challenge  and  inspired  the  initial  format  for  the 
HSI  Framework. 

1.  Establishing  a  Format  for  HSI  Assessment 

Assessment  of  HSI  required  a  novel  approach.  Aggregating  measures  had 
to  be  consolidated  across  seven  HSI  domains  and  then  presented  in  an 
integrated  and  understandable  manner.  A  variety  of  HF  and  HSI  frameworks, 
tools,  and  evaluation  schemes  were  analyzed,  including  those  discussed  in 
Chapter  II.  The  NASA  task-load  index  (TLX)  scale  presented  several  interesting 
qualities  pertinent  to  the  task  of  assessing  HSI.  TLX  was  developed  by  NASA  in 
the  1980s  for  assessing  human  workload  in  human-technology  systems  (Hart  & 
Staveland,  1988).  NASA’s  current  version  of  the  scale  is  shown  in  Figure  17. 


Figure  17.  NASA  TLX  scale  (electronic  version) 
(from  Wikipedia  commons,  2012) 
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The  TLX  scale  assesses  multiple  task  dimensions  within  a  given  context. 
Similarly,  the  goal  of  an  HSI  assessment  is  to  capture  the  contributions  of  each 
HSI  domain  and  assess  their  contribution  relative  to  total  system  performance. 
The  NASA  TLX  scale  demonstrated  that  this  type  of  multi-dimension  evaluation 
could  be  applied  successfully  (Hart,  2006).  This  provided  the  inspiration  for  the 
initial  HSI  assessment  framework. 

2.  Development  of  an  Initial  HSI  Assessment  Framework 

As  a  first  step  toward  constructing  an  evaluation  framework,  desired  end 
states  for  each  HSI  domain  were  adopted  from  the  HSI  FOM.  Definitions  were 
crafted  to  focus  the  assessment  of  each  HSI  domain,  similar  to  the  questions 
provided  for  NASA  TLX  performance  dimensions.  Initial  HSI  domain  definitions 
appear  in  Figure  20. 

The  initial  HSI  framework  borrowed  heavily  from  the  format  and  orientation 
of  NASA  TLX.  Initial  blackboard  prototypes  were  evolved  into  a  spreadsheet- 
based  framework,  as  seen  in  Figure  18.  HSI  domain  definitions  were  listed 
across  the  top  of  the  matrix,  with  rating  scales  in  the  columns  beneath  each 
domain.  A  data  field  was  provided  beneath  each  domain  definition  for 
practitioners  to  list  aggregating  measures  and  results. 

Next,  a  qualitative,  five-level  rating  scale  was  added  to  capture  the 
contribution  of  each  domain  to  overall  system  performance.  This  “total  system 
performance  implication”  (TSPI)  rates  the  extent  to  which  the  integration  of 
human  operators,  maintainers,  and  supporters  into  the  system  enhances  or 
degrades  the  total  performance.  The  TSPI  scale  operationalized  HSI 
performance  into  five  categories  (from  worst  to  best):  severe  degradation, 
moderate  degradation,  borderline  to  mild  degradation,  enhancement,  and 
optimizing,  as  seen  in  Figure  18. 
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Figure  18.  Initial  HSI  framework  (four  domains) 


Criteria  were  developed  to  assist  practitioners  in  assigning  a  total  system 
performance  implication  rating  based  on  the  aggregating  measures  for  each 
domain.  To  identify  language  capable  of  distinguishing  performance  within  and 
across  multiple  HSI  domains,  an  in-depth  review  of  HSI  standards  and  policies 
was  conducted.  Criteria  were  synthesized  from  a  wide  variety  of  standards 
including  MILSTD  1472-G,  NASA  STD-3000,  the  Air  Force  HSI  Handbook,  and 
the  U.S.  Army  MANPRINT  Handbook.  Potential  evaluation  categories  were 
developed  based  on  their  ability  to  apply  across  HSI  domains.  The  initial 
categories  were  identified  as  human-technology  balance  factor,  human  risk 
factor,  and  weight  of  contributing  evidence.  These  categories  and  their  rating 
criteria  are  shown  in  Table  5. 
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Balancing  of  Human  & 
Technological  Needs 

Human  Risk  Factor 

Weight  of  Contributing 
Evidence 

5 

Very  Good.  The  limitations  of 
human  users  are  fully 
understood  and  accounted  for 
across  the  system  for  the 
given  domain;  application  of 
technology  is  tailored  for 
optimal  leveraging  of  human 
capabilities  to  fulfill 
technology  constraints. 

Completely  Acceptable. 

The  system  places  humans 
users  are  well  within 
acceptable  limits  for  health, 
safety  and  welfare  for  the 
given  domain;  task  design, 
work  design, 

Validated  as  Positive. 

Domain  performance  meets 
or  exceeds  minimum 
thresholds,  benchmarks  and 
standards  confirmed  across 
the  spectrum  of  end  users, 
operating  modes  and 
conditions,  in  relevant 
environments. 

4 

Good.  The  limitations  of 
human  users  are  understood 
and  accounted  for  in  system 
design  for  the  given  domain; 
application  of  technology  is 
tailored  for  effective  use  of 
human  capabilities  to  fulfill 
technology  constraints. 

Reasonably  Acceptable. 

The  system  places  humans 
within  acceptable  limits  for 
health,  safety  and  welfare; 
human  functions,  work 
design  and  work  schedule 
contribute  to  effective  and 
efficient  and  safe  system 
performance. 

Verified  as  Positive. 

Domain  performance  meets 
or  is  predicted  to  meet 
minimum  thresholds, 
benchmarks  and  standards, 
verified  by  prototype  testing, 
high  fidelity  simulation,  or 
similarly  reliable  means. 

3 

Borderline.  The  limitations  of 
human  users  are  not  fully 
accounted  for  in  the  system 
design;  the  application  of 
technology  fails  to  leverage 
available  human  capabilities— 
or-  borders  on  exceeding 
human  capabilities. 

Borderline.  The  system 
places  humans  narrowly  or 
partially  within  acceptable 
limits  for  health,  safety  and 
welfare,  with  occasional 
excursions  from  safe  limits 
or  safe  work  practices. 

Inconclusive.  Domain 
performance  may  or  may  not 
meet  applicable  thresholds, 
benchmarks  and  standards; 
information  derived  from 
testing  or  simulation  is  not 
available  -or-  is  inadequate 
to  confirm  compliance. 

2 

Poor.  Human  capabilities  and 
limitations  are  poorly  matched 
to  system  aggregated  tasks 
and  workload;  technology 
unnecessarily  impedes, 
impairs  or  poorly  human 
capabilities  with  technology 
constraints. 

Unacceptable.  The  system 
places  humans  routinely 
placed  outside  of  acceptable 
limits  for  health,  safety  and 
welfare  for  the  given  domain; 
major  contribution  to 
program  risk. 

Verified  as  Negative. 

Domain  performance 
thresholds,  benchmarks  and 
standards  are  not  met  -or- 
domain  performance 
thresholds,  benchmarks  and 
standards  are  undefined, 
and  therefore  untested. 

1 

Very  Poor.  Human 
capabilities  and  limitations 
are  very  poorly  accounted  for 
in  the  system  design;  the 
application  of  technology 
unnecessarily  and 
unacceptably  impairs  human 
capabilities  to  fulfill 
technology  constraints. 

Extremely  Unacceptable. 

The  system  routinely  places 
humans  outside  of 
acceptable  limits  for  health, 
safety  and  welfare  for  the 
given  domain;  high  potential 
exists  for  mishap,  injury  or 
death. 

Validated  as  Negative. 

Domain  performance  has 
tested  below  threshold 
levels,  as  confirmed  with 
actual  system  users  in 
prototypes  or  in  high  fidelity 
simulation. 

Table  5.  Initial  HSI  framework  rating  scale  criteria 


The  content  and  compatibility  of  scales  across  HSI  domains  were  a 
subject  of  continuous  discussion  and  refinement  during  the  early  HSI  Framework 
development.  This  initial  HSI  Framework  and  rating  scale  established  a  starting 
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point.  Further  refinement  of  the  framework  required  feedback  from  current  HSI 
practitioners,  as  described  in  the  next  section. 

C.  STEP  THREE:  REFINING  THE  FRAMEWORK  WITH  HSI 

PRACTITIONERS 

The  initial  CHIEF  rating  approach  was  refined  during  a  series  of  four 
workshops  with  experienced  HSI  practitioners  from  the  U.S.  Coast  Guard  Human 
Systems  Integration  Division  (CG-1B3).  The  panel  consisted  of  six  active-duty 
and  civilian  HSI  personnel  assigned  to  CG-1B3.  Panel  members  represented  all 
HSI  domains  and  ranged  in  experience  from  0-4  to  0-5  (military),  and  GS-12  to 
GS-14  (civilian).  Practitioners  participated  remotely  from  CG-1B3  offices  located 
at  U.S.  Coast  Guard  Headquarters  in  Washington,  DC.  Participation  was  strictly 
voluntary,  and  interaction  was  documented  on  a  non-attribution  basis.  The 
workshop  plan  was  reviewed  by  the  NPS  Institutional  Research  Board  (IRB)  and 
was  determined  not  to  be  human  subject  research.  However,  the  workshop 
participants  were  accorded  the  rights  and  privileges  to  which  they  would  have 
been  entitled  had  it  been  deemed  human  subjects  research. 

The  four  workshops  were  hosted  by  NPS  via  virtual  meeting  software  over 
a  period  of  ten  weeks.  Participants  were  invited  to  join  a  virtual  collaboration 
worksite  where  workshop  presentation  material,  references,  and  participant 
uploads  were  made  accessible.  Sessions  were  jointly  moderated  by  the  author 
and  NPS  HSI  faculty  members.  Workshops  were  scheduled  according  to  CG- 
1B3  personnel  availability  and  averaged  approximately  two  hours  in  length.  A 
screenshot  of  the  workshop  collaboration  site  and  presentation  materials  for  all 
workshops  is  contained  in  the  appendices. 

1.  Workshop  One:  Introduction  and  Framework  Refinement 

The  first  workshop  introduced  participants  to  the  research,  processes,  and 
products  developed  for  this  thesis.  Six  participants  were  in  attendance  and  seven 
HSI  domains  were  represented.  Ground  rules,  objectives,  and  activities  for  the 
workshop  series  were  presented  for  discussion.  An  overview  of  the  HSI  Activity 
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Model  and  the  initial  HSI  Framework  were  presented,  followed  by  a  question  and 
answer  session. 

After  the  opening  presentation,  draft  HSI  domain  definitions  (Figure  18) 
were  presented.  Workshop  participants  were  prompted  with  a  scenario  in  which  a 
senior  acquisitions  official  had  requested  an  abbreviated  definition  of  their 
particular  HSI  domain,  containing  no  jargon  or  buzzwords.  Practitioners  were 
then  asked  to  refine  the  draft  HSI  domain  definitions. 

A  thirty-minute  breakout  session  was  conducted,  during  which  the 
practitioners  revised  the  initial  HSI  domain  definitions.  The  revised  definitions 
were  submitted  electronically  and  then  presented  to  the  group.  The  workshop 
concluded  after  consensus  had  been  achieved  and  plans  for  a  second  workshop 
were  discussed.  The  finalized  HSI  domain  definitions  appear  in  Figure  19. 


Manpower 

The  mixture  of  required  positions  is  balanced  for  the 
system's  aggregated  human  workload,  and  users  are  within 
acceptable  limits  of  endurance,  fatigue,  and  skills 

Personnel 

Criteria  are  sufficiently  established  to  identify,  train  and 
retain  personnel  capable  of  performing  at  or  above  the 
levels  required  by  the  system,  within  projected  manpower 
limitations. 

Performance 
Support  & 
Training 

The  instructional  system  design  develops  and  delivers  the 
human  performance  needed  for  efficient,  effective  and  safe 
system  operation  within  manpower,  personnel  and  system 
constraints 

Human 

Factors 

Engineering 

The  system  design  optimizes  the  capabilities  and 
limitations  of  human  operators,  maintainers,  and  support 
personnel  for  efficient,  effective,  suitable,  and  safe  system 
performance. 

Systems 
Safety  & 
Occupational 
Health 

The  system  safeguards  human  users  and  equipment  from 
acute  and  chronic  hazards  to  health  and  safety  to  enable 
efficient,  effective  and  safe  system  performance. 

Survivability 

The  system  safeguards  human  users  from  hazards,  and 
provides  efficient  and  effective  means  for  emergency 
egress  and  escape . 

Habitability 

The  system  configuration  and  environment  features 
support  effective  human  performance  by  providing  for  the 
comfort,  convenience,  and  quality  of  life  of  personnel. 

Figure  19.  Finalized  HSI  domain  definitions 
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2.  Workshop  Two:  Refinement  of  Rating  Scale  Criteria 

Workshop  Two  focused  on  refining  the  rating  scale  criteria  for  the  HSI 
Framework.  Four  of  the  Workshop  One  practitioners  were  in  attendance,  and  six 
HSI  domains  were  represented.  The  draft  framework,  proposed  measurement 
approach  (Figure  16),  a  proposed  workflow,  and  the  draft  rating  scale  criteria 
were  presented,  followed  by  an  extensive  question  and  answer  session. 
Practitioner  input  focused  on  the  perceived  strengths  and  weakness  of  the 
aforementioned  materials  based  on  their  experience. 

Practitioners  identified  three  central  concerns  stemming  from  the  initial 
HSI  Framework.  First,  the  use  of  a  five-level  rating  scale  (Table  5)  was 
considered  excessive.  A  main  point  of  contention  was  that  the  inclusion  of  two 
levels  of  performance  above  “acceptable”  did  not  align  favorably  with  current 
acquisition  practices,  which  are  based  on  objective  and  threshold  values  (for 
additional  description  of  these  values,  see  the  Defense  Acquisition  Guidebook 
(DAG)  2011,  Section  2.1.1).  Second,  the  rating  scale  criteria  (i.e.,  defined  scale 
levels)  were  viewed  by  participants  as  containing  too  much  detail,  inviting  overly 
subjective  interpretation.  And  third,  ratings  across  categories  were  evaluated  as 
not  sufficiently  synchronized  with  the  TSPI  scale  (Figure  18).  In  particular, 
practitioners  felt  that  they  could  not  clearly  differentiate  between  acceptable  and 
non-acceptable  HSI  performance,  given  the  initial  rating  scale. 

Following  the  opening  discussion,  practitioners  were  provided  draft  rating 
scales  and  asked  to  revise  the  wording.  Practitioners  were  informed  they  would 
be  asked  to  rate  a  current  system  acquisition  for  an  upcoming  management 
briefing  and  to  defend  their  HSI  ratings  based  on  the  scale  criteria.  A  thirty- 
minute  breakout  session  was  then  held  for  discussion  and  analysis.  The 
workshop  adjourned  with  a  request  from  the  author  for  continued  offline 
discussions  to  resolve  the  concerns  raised.  Recommended  modifications  were 
received  from  CG-1B3  within  several  days.  Practitioners  recommended  the 
following  adjustments: 


46 


•  Rating  scales  should  remain  at  five  levels  to  avoid  skewing  the 
scale  toward  suboptimal  HSI  performance  levels. 

•  Use  of  the  terms  “threshold”  and  “objective”  should  be  omitted  in 
favor  of  more  universal,  acquisition-independent  language. 

•  Scale  wording  should  be  aligned  such  that  the  minimum  level  of 
acceptability  occurs  between  levels  two  and  three. 

•  The  “Weight  of  Contributing  Evidence”  category  should  be 
separated  from  direct  rating  scales  and  implemented  separately. 

The  HSI  Framework  was  reformatted  and  refined  based  on  adjustments 
requested  by  the  practitioners.  Refinements  included  altering  the  scale  language 
to  harmonize  rating  categories,  removing  intermediary  scale  criteria  for  simplicity, 
and  isolating  and  renaming  the  “Weight  of  Contributing  Evidence”  category  as 
the  “Acquisition  Performance  Factor.”  It  was  requested  that  this  factor  be  added 
after  the  domain  had  been  scored.  An  “Accommodation  Factor”  category  was 
introduced  to  supplement  the  other  scales  based  on  its  applicability  across 
domains.  The  reformatted  rating  scales  and  criteria  are  presented  in  Table  6.  The 
acquisition  performance  rating  scale  is  presented  in  Table  7. 
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Balancing  of  Human  & 
Technological  Needs 

Human  Risk  Factor 

Weight  of  Contributing 
Evidence 

5 

Very  Good.  The  limitations  of 
human  users  are  fully 
understood  and  accounted  for 
across  the  system  for  the  given 
domain;  application  of 
technology  is  tailored  for  optimal 
leveraging  of  human  capabilities 
to  fulfill  technology  constraints. 

Completely  Acceptable.  The 

system  places  humans  users 
are  well  within  acceptable 
limits  for  health,  safety  and 
welfare  for  the  given  domain; 
task  design,  work  design, 

Validated  as  Positive. 

Domain  performance  meets  or 
exceeds  minimum  thresholds, 
benchmarks  and  standards 
confirmed  across  the  spectrum 
of  end  users,  operating  modes 
and  conditions,  in  relevant 
environments. 

4 

Good.  The  limitations  of  human 
users  are  understood  and 
accounted  for  in  system  design 
for  the  given  domain; 
application  of  technology  is 
tailored  for  effective  use  of 
human  capabilities  to  fulfill 
technology  constraints. 

Reasonably  Acceptable.  The 

system  places  humans  within 
acceptable  limits  for  health, 
safety  and  welfare;  human 
functions,  work  design  and 
work  schedule  contribute  to 
effective  and  efficient  and  safe 
system  performance. 

Verified  as  Positive.  Domain 
performance  meets  or  is 
predicted  to  meet  minimum 
thresholds,  benchmarks  and 
standards,  verified  by 
prototype  testing,  high  fidelity 
simulation,  or  similarly  reliable 
means. 

3 

Borderline.  The  limitations  of 
human  users  are  not  fully 
accounted  for  in  the  system 
design;  the  application  of 
technology  fails  to  leverage 
available  human  capabilities-or- 
borders  on  exceeding  human 
capabilities. 

Borderline.  The  system  places 
humans  narrowly  or  partially 
within  acceptable  limits  for 
health,  safety  and  welfare,  with 
occasional  excursions  from 
safe  limits  or  safe  work 
practices. 

Inconclusive.  Domain 
performance  may  or  may  not 
meet  applicable  thresholds, 
benchmarks  and  standards; 
information  derived  from 
testing  or  simulation  is  not 
available  -or-  is  inadequate  to 
confirm  compliance. 

2 

Poor.  Human  capabilities  and 
limitations  are  poorly  matched 
to  system  aggregated  tasks  and 
workload;  technology 
unnecessarily  impedes,  impairs 
or  poorly  human  capabilities 
with  technology  constraints. 

Unacceptable.  The  system 
places  humans  routinely 
outside  of  acceptable  limits  for 
health,  safety  and  welfare  for 
the  given  domain;  major 
contribution  to  program  risk. 

Verified  as  Negative.  Domain 
performance  thresholds, 
benchmarks  and  standards  are 
not  met-or-domain 
performance  thresholds, 
benchmarks  and  standards  are 
undefined,  and  therefore 
untested. 

1 

Very  Poor.  Human  capabilities 
and  limitations  are  very  poorly 
accounted  for  in  the  system 
design;  the  application  of 
technology  unnecessarily  and 
unacceptably  impairs  human 
capabilities  to  fulfill  technology 
constraints. 

Extremely  Unacceptable.  The 

system  routinely  places 
humans  outside  of  acceptable 
limits  for  health,  safety  and 
welfare  for  the  given  domain; 
high  potential  exists  for 
mishap,  injury  or  death. 

Validated  as  Negative. 

Domain  performance  has 
tested  below  threshold  levels, 
as  confirmed  with  actual 
system  users  in  prototypes  or 
in  high  fidelity  simulation. 

Table  6.  Finalized  CHIEF  rating  scale  criteria 


Acquisition  Progression  Factor 
(specific  to  each  HSI  domain) 

Behind 

The  quantity  or  quality  of 
contributing  evidence  leading 
to  the  domain  score  is  less 
than  expected/required  for  the 
current  acquisition  phase. 

Concurrent 

The  quantity  and  quality  of 
contributing  evidence  is 
commensurate  with  the 
acquisition  phase. 

Ahead 

The  quantity  and  quality  of 
contributing  evidence  is  more 
than  expected/required  for  the 
current  acquisition  phase. 

Table  7.  Acquisitions  performance  factor  (APF) 
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3.  Workshop  Three:  Framework  Finalization 

Workshop  Three  was  aimed  at  finalizing  the  rating-scale  criteria  and  the 
design  of  the  HSI  Framework  scoring  matrix.  Six  of  the  Workshop  One 
practitioners  were  in  attendance,  and  five  HSI  domains  were  represented. 
Workshop  participants  were  presented  with  the  revised  rating  scale  definitions, 
including  the  “Accommodation  Factor”  category.  The  reformatted  rating  criteria 
met  with  broad  consensus. 

Candidate  designs  of  the  HSI  Framework  scoring  matrix  were  then 
presented.  Alternatives  included  two  versions  with  vertical  performance  scales 
(similar  to  TLX),  and  one  with  a  horizontal  performance  scale.  Practitioners  were 
asked  to  select  the  candidate  they  considered  most  effective  at  conveying  HSI 
assessment  results.  The  horizontal  format  (Figure  20)  was  selected  by  the 
practitioners. 


Project  ID 

Need 

Analyze  & 
Select 

Obtain 

Produce  & 
Deploy 

_ 

(-)Very  pocr 

(-)Fbor 

Minimally 

Acceptable 

(+)Good 

(++)Very 

Good 

(-)  Bdremely 
Lhacosptable 

<-) 

Unacceptable 

Minimally 

Acceptable 

(+)  Fteascnabfy 
Axeptable 

(++) 

Ccrrpletely 

Axectable 

(-)Very  poor 

(-)  Fbor 

Mrimaly 

■Acceptable 

(+)Good 

(++)Very 

Gocd 

Figure  20.  Selected  scoring  matrix  with  example  ratings 
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The  workshop  concluded  after  the  practitioners  had  nominated  a  test  case 
that  would  be  used  in  the  final  workshop  to  validate  the  HSI  Framework.  Past, 
current,  and  future  acquisition  programs  were  considered  as  candidates.  A 
recently  fielded  system,  the  Coast  Guard  Fast  Response  Cutter  (FRC)  (Figure 
21)  was  selected  for  the  case  study.  A  poll  revealed  that  all  participants  had 
recent  work  experience  within  the  FRC  acquisition.  The  recently  completed 
Production  Readiness  Review  (PRR)  would  serve  as  a  scenario  for  the  HSI 
Framework  rating  (for  more  information  on  the  engineering  review  process,  see 
the  DHS  Instruction/Guidebook  102-01-001,  Appendix  B  -  Systems  Engineering 
Lifecycle). 


Figure  21 .  Coast  Guard  Fast  Response  Cutter  (FRC)  (from  U.S.  Coast 
Guard  heartland.coastguard.DODIive.mil) 

4.  Workshop  Four:  HSI  Framework  Test  Case 

Workshop  Four  employed  the  HSI  Framework  to  conduct  a  simulated  HSI 
assessment  of  the  FRC  acquisition  program  prior  to  the  PRR.  Four  out  of  the  six 
original  practitioners  were  in  attendance,  and  six  HSI  domains  were  represented. 
The  workshop  focused  on  evaluating  the  usability  of  the  HSI  framework  and 
establishing  a  preferred  scoring  method. 
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The  workshop  began  with  a  briefing  on  the  test  scenario.  Practitioners 
were  asked  to  draw  on  their  knowledge  of  specific  HSI  measures  and  test  data  at 
the  time  of  the  PPR  to  develop  HSI  Framework  ratings  and  then  present  domain 
scores  and  justifications  to  the  group.  An  example  workflow  for  the  HSI 
Framework  was  presented  by  the  moderators.  A  thirty-minute  breakout  session 
was  held  to  develop  ratings  for  each  HSI  domain.  CG-1B3  leaders  compiled 
individual  domain  ratings  for  the  FRC.  Practitioners  then  presented  justifications 
for  each  rating  based  on  the  HSI  Framework  rating  scales  and  criteria. 

With  HSI  Framework  domain  scores  established,  the  desired  method  for 
representing  those  scores  became  the  focus  of  the  discussion.  The  merits  of 
various  numerical  approaches  were  presented  by  the  domain  practitioners  and 
HSI  managers.  Several  key  areas  were  addressed,  including  the  appropriate 
level  of  numerical  accuracy  (i.e.,  number  of  decimal  places),  desired  technique 
for  numerical  averaging,  and  the  possibility  of  weighting  the  domains.  The 
workshop  concluded  with  consensus  on  the  following  measurement  approach: 

•  Numerical  scoring  would  be  limited  to  whole  numbers,  consistent 
with  the  level  of  precision  afforded  by  HSI  measures. 

•  The  Acquisition  Performance  Factor  (APF)  would  be  assigned  a 
plus  or  minus  symbol  (+/-),  in  order  to  indicate  the  trend  of 
performance  in  each  domain  without  affecting  the  numerical  score. 

•  Where  possible,  a  numerical  average  should  be  avoided  in  favor  of 
displaying  a  complete  score  across  all  domains. 

•  When  an  average  was  desired  (i.e.,  representation  of  all  HSI 
domains  scores  was  not  possible),  scores  falling  below  the 
acceptable  level  should  also  be  indicated. 

•  The  final  framework  should  include  the  ability  to  declare  selected 
domains  to  be  not  applicable  for  specific  projects. 

D.  FINAL  VERSION  OF  THE  CHIEF 

The  final  version  of  the  HSI  Framework  assembles  all  the  components 
refined  during  the  four  workshops:  the  scoring  matrix,  the  finalized  domain 
definitions,  and  the  rating  scales  with  guiding  criteria.  We  refer  to  these 
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collectively  as  the  Comprehensive  Human  Integration  Evaluation  Framework  (or 
CHIEF)  rating  system.  This  rating  system  can  be  used  to  assess  HSI  efficacy 
across  HSI  domains  in  any  system  acquisition  program  (Figure  22). 


Occupational  Health 
Survivability 


Figure  22.  Final  CHIEF  components 


The  final  version  of  the  CHIEF  rating  system  (at  the  top  of  Figure  22) 
includes  other  refinements  from  the  final  HSI  Framework  workshop.  An  indicator 
bar  to  display  the  current  phase  of  acquisition  (top)  and  information  fields  for  the 
author  and  occasion  for  the  rating  (bottom  left)  were  incorporated  into  the  final 
design.  Separate  fields  were  included  for  the  HSI  domain  rating  and  the 
acquisition  performance  factor. 
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The  final  version  of  the  CHIEF  rating  system  is  configured  as  a  “rolled-up” 
view,  removing  background  information  such  as  HSI  domain  descriptions  and  the 
aggregating  measures  seen  in  the  initial  HSI  Framework.  As  determined  in  the 
final  workshop,  these  fields  remain  hidden  and  can  be  is  implemented  as  display- 
on-demand  information  when  required.  The  rating  scale  criteria,  aggregated 
measures,  and  APF  scale  are  similarly  hidden. 

As  will  be  discussed  in  the  final  chapter,  incorporating  on-demand  display 
of  information  was  considered  highly  desirable  for  future  versions  of  CHIEF. 

1.  Suggested  CHIEF  Workflow 

Figure  23  shows  the  notional  workflow  for  developing  CHIEF  ratings,  as 
conceptualized  by  the  author  and  NPS  HSI  faculty,  and  validated  by  practitioners 
during  the  final  HSI  Framework  refinement  workshop. 


^Ongoing  Major  Acquisition  j 4 
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initiates 

START _ ^ _ 


„  informs - ^  6.  Overall  HSI  Rating  j 


1.  Assessment  of  the  Manpower  HSI 
domain  is  required 
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2.  Practitioner  identifies 
relevant  measures  &  gathers  results 
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Legacy  System 
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Figure  23.  Example  CHIEF  workflow 


The  notional  workflow  for  the  development  of  CHIEF  ratings  can  be 
summarized  in  six  steps: 


•  Step  One:  The  practitioner  is  assigned  to  evaluate  HSI 

performance  in  the  selected  domain(s)  for  a  given  system.  The 
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practitioner  then  reviews  the  consolidated  domain  definition  to 
guide  assessment. 

•  Step  Two:  The  practitioner  applies  his  or  her  experience  to  identify 
aggregating  measures  appropriate  for  the  system;  data  from  these 
measures  are  then  collected  for  analysis. 

•  Step  Three:  The  results  from  the  data  collected  are  consolidated 
and  assessed  using  the  CHIEF  rating  scales  and  criteria;  the 
practitioner  applies  his  or  her  knowledge  to  determine  the  impact  of 
the  selected  domain(s)  on  total  system  performance  (e.g., 
manpower  in  Figure  22). 

•  Step  Four:  Once  the  CHIEF  domain  rating  has  been  assessed,  the 
results  and  a  remediation  strategy  (as  required)  are  reviewed  by  a 
competent  authority.  Issues  are  resolved  (if  possible)  at  the 
appropriate  organizational  level. 

•  Step  Five:  The  CHIEF  domain  rating  is  assigned,  including  a 
summary  of  aggregated  measures,  source  documentation,  and  a 
resolution  strategy  for  discrepancies  (as  required). 

•  Step  Six:  The  individual  CHIEF  HSI  domain  ratings  are 
consolidated  with  other  domains  for  a  comprehensive  CHIEF 
assessment. 

In  addition  to  reducing  the  potential  for  error,  a  standardized  CHIEF 
workflow  offers  other  benefits.  Administrative  process  controls  can  be 
implemented  at  any  step,  offering  senior  HSI  practitioners  and  acquisitions 
managers  insight  into  HSI  activities  as  they  evolve.  A  standard  workflow  also 
facilitates  process  automation  through  software,  as  discussed  in  the  next 
chapter. 
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V.  SUMMARY  AND  CONCLUSION 


A.  SUMMARY 

This  thesis  documents  the  development  of  a  simplified  framework  for 
understanding  and  assessing  HSI  across  diverse  disciplines  of  acquisition. 

The  proposed  HSI  Activity  Model  was  developed  to  conceptualize  HSI 
activity  in  military  acquisition.  Established  human  factors  and  human  computer 
interaction  theory  was  studied  and  applied  to  develop  a  concise  view  of  HSI  in 
action  during  military  system  acquisition.  In  this  model,  we  summarized  the  core 
activity  of  HSI  as  the  balancing  of  human  capabilities  and  limitations  with  the 
affordances  and  constraints  presented  by  system  technology  to  accomplish 
system  objectives. 

Next,  we  developed  the  CHIEF  rating  system  to  provide  a  method  for 
assessing  HSI  during  acquisition.  Existing  system  assessment  measures, 
including  TRL,  were  studied  to  determine  criteria  for  a  successful  HSI 
measurement  framework.  A  measurement  approach,  rating  scales  and  criteria, 
and  a  consolidated  scoring  matrix  were  then  developed  for  the  framework  in 
cooperation  with  HSI  faculty  from  the  Naval  Postgraduate  School.  This  initial 
version  of  CHIEF  was  refined  during  four  workshops  with  Coast  Guard  HSI 
practitioners.  As  a  culminating  exercise,  practitioners  employed  CHIEF  to 
evaluate  a  major  USCG  acquisition  program.  The  usability  of  the  framework, 
notional  workflow,  and  preferred  scoring  method  were  then  refined. 

Our  research  produced  a  usable  framework,  but  revealed  a  number  of 
necessary  assumptions,  limitations,  and  practical  considerations  that  must  be 
taken  into  account.  These  are  discussed  in  the  following  sections. 
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B.  EXAMINING  CHIEF  ASSUMPTIONS  AND  LIMITATIONS 

The  research,  discussions,  and  scenarios  carried  out  in  CHIEF’S 
development  required  a  number  of  assumptions  and  revealed  some  potential 
limitations.  The  following  emerged  as  common  themes: 

1.  Subject  Matter  Expertise  Is  Essential 

Effective  use  of  CHIEF  depends  on  the  expertise  of  HSI  practitioners.  As 
seen  in  the  proposed  workflow  (Figure  23),  a  CHIEF  evaluation  requires  in-depth 
knowledge,  skills,  and  abilities  in  the  HSI  domain.  This  becomes  most  evident 
during  selection  of  aggregating  measures  and  during  interpretation  of 
performance  results  using  CHIEF  rating  scales.  Developing  an  overall  CHIEF 
rating  across  domains  requires  knowledge  of,  and  experience  with,  overarching 
HSI  policies  and  procedures.  Without  the  required  practitioner  expertise,  CHIEF 
ratings  may  very  well  be  inaccurate. 

2.  Aggregating  Measures  Are  Pivotal 

The  accuracy  of  CHIEF  is  predicated  on  relevant,  valid,  and  reliable  HSI 
TTAMs.  Measures  must  be  available  to  practitioners  in  areas  that  require 
measurement;  program  resources  must  also  be  available  for  aggregating 
measures  to  be  planned  for  and  collected  in  a  timely  fashion  during  acquisition. 
Establishing  reliable  measures  is  essential  to  the  successful  implementation  of 
CHIEF  as  the  standard  for  assessing  the  extent  to  which  human  and 
technological  components  form  a  healthy,  synergistic  bond  that  will  achieve  total 
system  objectives. 

3.  Recognize  Statistical  and  Representational  Limitations 

The  system-wide  HSI  assessment  proposed  in  CHIEF  will  require  a  wide 
variety  of  measures.  Alignment  of  composite  aggregating  measures  across 
CHIEF  ratings  scales  will  also  require  subjective  interpretation,  as  noted  earlier. 
For  these  reasons,  data  derived  from  CHIEF  should  be  regarded  as  ordinal  data 
(Stangor,  2010).  The  numerical  scores  generated  by  CHIEF  imply  the  ability  to 
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calculate  an  overall  HSI  average  for  an  acquisition  program,  but  this  temptation 
should  be  resisted.  Since  no  weighting  scale  is  applied  during  the  CHIEF  rating 
process,  a  straightforward  averaging  of  scores  suggests  equal  weighting  of  all 
domains,  regardless  of  system  realities.  Using  an  un-weighted  mean  when  all 
domains  are  not  equal  will  result  in  a  skewed  perception  of  HSI  efficacy  in  the 
acquisition  program. 

CHIEF  was  conceived  for  conveying  HSI  understanding  during  key 
acquisition  events.  Thus,  CHIEF  results  are  expected  to  share  limited  space  with 
other  acquisition  measures  and  information.  Displaying  individual  CHIEF  domain 
scores  and  supporting  information  may  not  be  feasible.  In  circumstances  where  a 
composite  HSI  assessment  is  desired,  using  an  average  score  based  on  the  total 
system  performance  implication  scale  is  suggested,  supplemented  with  an 
indication  of  all  domains  rated  below  the  acceptable  level  of  performance  (TSPI 
less  than  three).  This  will  avoid  underperforming  domain  scores  from  being 
obscured  during  averaging. 

C.  RECOMMENDED  DIRECTIONS  FOR  FUTURE  RESEARCH 

The  development  of  CHIEF  was  limited  by  the  time  and  resources 
available  for  this  thesis.  Though  functional,  the  elements  of  CHIEF  require  further 
refinement.  Several  promising  areas  for  continued  research  are  discussed  below. 

1.  Continue  Developing  the  CHIEF  Software 

Developing  CHIEF  to  its  full  potential  will  require  the  framework  to  be 
placed  into  a  usable  format.  The  current  spreadsheet-based  version  of  CHIEF 
requires  extensive  user  manipulation  and  has  limited  input  and  output.  This  was 
sufficient  for  research  and  rapid  prototyping,  but  not  for  adoption  as  an  accepted 
HSI  business  practice.  Development  of  specific,  dedicated  software  was  beyond 
the  scope  of  this  effort.  However,  CHIEF’S  development  and  refinement  revealed 
many  desirable  attributes  for  future  software  application  (see  Appendix  C). 
Overall,  the  workflow  and  design  of  Dl’s  SHARE  software  tool  offers  an  excellent 

starting  point  for  developing  a  CHIEF  software  architecture. 
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2.  Refine  CHIEF  Rating  Scale  Criteria 

One  of  the  most  challenging  aspects  of  developing  CHIEF  was 
establishment  of  rating  scale  criteria.  Criteria  needed  to  be  precise  enough  to 
discriminate  between  levels  of  HSI  performance  without  extraneous  detail. 
Criteria  also  needed  to  be  compatible  with  aggregating  measures  across  multiple 
HSI  domains.  The  scales  and  criteria  presented  in  this  thesis  were  validated  in 
the  final  workshop  scenario,  but  with  a  relatively  small  group  of  practitioners  in  a 
controlled  scenario.  Additional  subscales  and  criteria  will  likely  be  required  to  suit 
a  broader  range  of  practitioners,  and  more  diverse  evaluation  scenarios. 

3.  Identify  Suitable  Aggregating  Measures 

As  proposed  in  Chapter  IV,  aggregating  measures  evaluate  as  they 
interact  with  system  components  in  relevant  work  context.  The  broad  range  of 
systems  in  modern  military  acquisition  means  that  this  context  will  vary  widely. 
Enterprise  software  systems,  major  cutters,  and  unmanned  aircraft,  for  instance, 
represent  very  different  operational  contexts  and  may  require  a  different  suite  of 
HSI  TTAM  to  capture  desired  measures.  Using  the  CHIEF  framework  will  require 
an  organization  to  identify  relevant  measures  for  each  specific  type  of  system. 

D.  CONCLUSION 

Carrying  out  effective  HSI  remains  a  critical  challenge  for  the  USCG,  DHS, 
and  DOD.  This  thesis  breaks  new  ground  to  establish  useful  metrics  towards  this 
end.  As  the  current  body  of  research  reveals,  achieving  the  necessary  integration 
of  humans  and  technology  is  a  complex  and  challenging  task.  The  history  of  TRL 
suggests  that  decades  may  pass  before  human-centered  system  measures  are 
embraced  and  fully  incorporated  into  federal  acquisitions.  The  best  way  to 
accelerate  this  timeline  is  to  incorporate  CHIEF  into  acquisition  policy  and  require 
the  use  of  CHIEF  for  each  milestone  decision.  Similar  to  TRL  ratings,  if  an 
acquisition  program  receives  an  unsatisfactory  CHIEF  rating,  the  program  cannot 
proceed  to  the  next  phase  until  the  integration  of  humans  and  technology  is 
deemed  acceptable. 
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We  hope  that  this  research  and  the  products  offered  will  expand 
awareness  of  how  critical  the  human  component  is  to  modern  systems 
acquisition.  Our  collective  understanding  of  human-technology  integration  (or 
lack  thereof)  will  determine  the  survival  of  our  service  members  and  shape  our 
operational  fortunes. 
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APPENDIX  A  -  HSI  FIGURE  OF  MERIT 


PART  ONE 


Domain 

Manpower 

Personnel 

Performance  Support  & 
Training 

The  correct  number  and  mix  of 
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"O 

Ol 
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C 
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DAG  Chapter  6.3 
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PART  TWO 


Domain 

Human  Factors  Engineering 

Anthropometry 

Ergonomics 

Cognitive 

Engineering 

Macro  Ergonomics 

Desired  State 

Physical  characteristics, 
capabilities  and 
limitations  of  the 
expected  user  population 
are  established;  physical 
interfaces  including 
seating,  racks, 
workstations,  worksites, 
accesses  and  touch  points 
accommodate  the  full 
user  population. 

Functional  interfaces 
including  controls, 
displays,  workstations, 
worksites,  accesses, 
labeling  and  touch 
points  are  configured 
for  effective,  efficient 
and  safe  operation 
given  the  expected 
user  population. 

Cognitive  interfaces, 
processes  and 
information  flows 
are  optimized  for 
end-user  cognitive 
abilities;  end-user 
cognitive  resources 
including  memory, 
attention  and 
decision  making 
anchor  the  system 
design. 

Cooperative 
interfaces  are 
optimized  to  enable 
and  enhance  team 
performance, 
collaboration  and 
communication; 
organizational 
interfaces  are 

reflective  of  the 
projected  operating 
conditions  and 
deployment  scheme 
of  the  system. 

Design  Concerns 

User  physical 
characteristics, 
proportions  and 
dimensions, 

anthropometric  surveys, 
physical  strength,  range  of 
motion,  range  of 
perception;  accessibility, 
adjustability, 

configurability,  modularity 

Reaction  time, 
attention,  working 
memory;  display 
placement,  information 
management, 
eguipment 
arrangement,  color 
coding,  gestalt 
principles,  labeling, 
grouping  of  controls, 
ease  of  use,  usability, 
task  load  index 

Attention  cues, 
cognitive  workload, 
decision  rules, 
decision  support 
systems,  situation 
awareness,  mental 
models,  cognitive 
skills  and  attitudes, 
warnings  and 
alarms,  function 
allocation  (humans 
vs.  automation) 

Job  design,  operating 
policies  and 
procedures,  watch 
floor  and  workstation 
arrangement; 

communication 
scheme,  watch  team, 
management 
interfaces,  crew/team 
coordination,  shared 
mental  models. 

Undesired  State 

Capabilities  and 
limitations  of  the 
expected  user  population 
are  not  fully  understood  & 
applied;  tasks  and  work 
areas  are  inadequately 
arranged,  or  are  not 
configurable  to 
accommodate  the  full 
user  population. 

Physical  and  virtual 
interfaces  are  not 
configured  for 
effective,  efficient  and 
safe  operation; 
interfaces,  controls  or 
other  touch  points  do 
not  recognize  and 
incorporate  physical 
capabilities  and 
limitations  of  expected 

users. 

Cognitive  workload, 
processes  and 
information  flows 
are  incompatible 
with,  or  exceed  end- 
user  cognitive 
capabilities; 
functional  allocation 
(automation)  poorly 
balances  human 
cognitive  abilities 
and  system 
capabilities. 

Cooperative 
interfaces  are 
optimized  for  team 
performance, 
collaboration,  and 
inter-user 
communication; 
organizational 
interfaces  reflect 
system  operating 
conditions, 
constraints  and 

limitations. 
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PART  THREE 


Domain 

Systems  Safety  & 
Occupational  Health 

Habitability 

Survivability 

Desired  State 

The  full  extent  of  potential 
acute  and  chronic  hazards  to 
personnel  are  fully  understood; 
hazards  are  reduced, 
eliminated  or  mitigated 
commensurate  with  expected 
operations  to  minimize  risk  of 
acute  and  chronic  injuries  to 
the  expected  user  population. 

Living  and  working  conditions 
are  conducive  to  sustaining  the 
morale,  safety,  health,  and 
comfort  of  the  user 
population;  living  and  working 
conditions  enhance  personnel 
effectiveness  and  mission 
accomplishment,  and 
encourage  recruitment  and 
retention. 

System  is  configured  to  reduce 
the  risk  of  fratricide,  detection, 
and  the  probability  of  being 
attacked  commensurate  with 
projected  operations;  users  are 
able  to  withstand  hostile 
environments  without  aborting 
the  mission  or  acute  chronic 
illness,  disability,  or  death. 

Design  Concerns 

Walking/working  surfaces, 
work  at  heights,  pressure 
extremes,  lock-out/tag-out,  fire 
hazards,  and  explosive  risk; 
shock  hazards,  noise,  light, 
vibration,  chemical  exposure, 
radiological  exposure, 
repetitive  motion; 
psychological  risk  factors 

Lighting,  space,  ventilation, 
and  sanitation;  noise  and 
temperature  control;  ship 
motion;  religious,  medical,  and 
food  services  guality  and 
availability;  berthing,  bathing, 
and  personal  hygiene,  non- 
operational  connectivity 

Crew  compartment  integrity, 
egress  capability, 
emergency/rescue  access; 
armor,  ballistic  protection,  fire 
protection,  fire  suppression, 
decontamination  capability; 
fratricide,  detectability, 
resilience,  reparability 

Undesired  State 

The  system  fails  to  reduce, 
eliminate  or  mitigate 
personnel  hazards  to  an 
acceptable  level  for  expected 
operations;  system  is  not 
adequately  configured  to 
minimize  risk  of  acute  and 
chronic  injuries  to  the 
expected  user  population. 

Living  or  working  conditions 
inhibit  the  morale,  safety, 
health  and/or  comfort  of 
system  end-users.  Living  or 
working  conditions  diminish 
personnel  effectiveness  and 
mission  accomplishment, 
and/or  diminish  recruitment 
and  retention. 

System  fails  to  adequately 
reduce  the  risk  of  fratricide, 
detection,  and  the  probability 
of  being  attacked  in  projected 
operations;  users  are  unable  to 
withstand  hostile 
environments  without  aborting 
the  mission  and/or  suffering 
acute  chronic  illness  or  injury. 
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APPENDIX  B  -  WORKSHOP  MATERIALS 


NPS  COLLABORATION  SITE  SCREENSHOT 
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Naval  Postgraduate  School 
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WORKSHOP  ONE  PRESENTATION 


*  * 

Development  of  a  Comprehensive  Human 

Integration  Evaluation  Framework  (CHIEF) 


Workshop  1:  Intro  &  Domain  Definitions 
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HSI  in  a  Systems  Acquisition  Context 
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NASA/DOD  Technology  Readiness  Level 
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•  • :  •  *  ,► /t  .  ' 


* 


Building  a  Comprehensive 
Human  Integration 
Evaluation  Framework 
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CHIEF 


Task  Load 
Index  (TLX) 


>  Itfnmg 


CHIEF-  HFE  Example 


agement  Indicators 


Aggregated  Measures 


1  \  v<s*.i.ty  f 

contros  \ 

asp-ays 


First  Order  Effects 


Ml_STD  6S2-E  J 


MILSTD  1472-G 


Policies  &  Standards 
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Definitions  +  Aggregated  Measures 


* 


Personnel  Performance  Support  ft  Training  Human  Factor*  Engineering 

Oeeni  pro  luAoenfy  etfifitilej  FiiflTCftona  system  dngt  dnetopi  The  ijilern  design  Polancos  me 
to  idanMy.  Iran  and /scan  and  detvers  human  apMudes  needed  capaMbes  and  tmcaeonso/hianan 

pincntr  cipitiii  if  Apr  jfltomt  »nJ  i >li  ijifom  m#p  m*cTi  rr»  ofibnJyic#J  tncf 

Of  Sbov*  ths  Isvsit  rsqufsd  by  ops' stem  mrtho  m*npcm*f  sod  coo»trs**s  oftseboedogy for 

system  Mthf)  prOfGCtod  p#rjono#/  tmisboos  offcctrvp  sod  tsfs  iyttsoi 


m 


» Entetod  /  OWcer  Spsoatoes? 


e.  Operaaonaf) 
i  Ratio 


Aggregated  Measures 


FUgn  F  dewy  Task  SenUMon 
fcWlwinili  AnCfropometnc  Amf)M 
Coofwr-Morpor  Rjbng 
ift*  Asutywt  Tool  Scot# 


Measures  that  aggregate  policies,  standards  and  1K  order  effects 


Definitions,  Measures 
+  Rating  Scale 


(for  workshop  2):  Is  the  given 
domain  contributing  to  or 
hindering  overall  system 
performance? 
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Rating  Scale-  Expanded  View 


Questions 

How  does  the  system 
treat  humans? 


How  well  does  the 
system  balance 
technological  versus 
human  needs? 


How  certain  are  ive? 
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»  4 


You  are  approached  by  a  senior  acquisition  executive  (CAE 
or  similar),  who  wants  to  get  smarter  on  HSI.  They  ask  you: 

•  Could  you  send  me  a  1*2  sentence  summary  of  the 

_ [domain]  of  HSI  work.. .summarized  in  your  own 

words,  without  buzzwords  or  jargon?  Atthe  center  of  it  all, 
what  are  you  trying  to  accomplish  in _ [domain]? 

Your  excellent  staff  has  provided  you  with  a  draft.  Send 
your  edits  via  email. 


Workshop  Exercise  1: 
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WORKSHOP  TWO  PRESENTATION 


* 

Development  of  a  Comprehensive  Human 
Integration  Evaluation  Framework  (CHIEF) 
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HSI  in  a  Systems  Acquisition  Context 
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Workshop  2  Goals 

0  Refine  The  Scale  And  Rating  Criteria 

□  Rating  Scale:  Do  we  have  the  right  number  of 
categories? 

□  Scale  Considerations:  Are  we  answering  the  key 
questions? 

□  Criteria:  Do  we  have  the  right  scale  criteria? 

Desired  End-state:  Give  CG-1B3  practitioners  a  credible 
and  reliable  tool  to  advance  human  design  priorities. 


■M?  Example  of  Measuring  HSI: 

Nesting  of  standards  during  design  of  a  control  console 


gregated  Measure 


Tsw  i.'ooeung  ~au 
&  snuaw  non 


L030  C 

(tlx)  nan 


Cocper- 

Marper  scale  '  Collective 

Performance  Measures 


Operawvg  I  Physical 

Properties  I  L9yO« 


umnance 

Scneme 


WLSTD  SC-E 
MISTD  1472-G 


v  \ASA-STD-3031 


System  Attributes 


Policies  &  Standards 
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WORKSHOP  THREE  PRESENTATION 
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problem 

statement 

The  acquisitions  community  would  benefit  from 
a  common  framework  to  understand,  assess 
and  communicate  HSI. 
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•  Minimal  acceptability  at  rating  of  3 

•  Middle  definitions  removed 

•  Ratings  earned  by  fulfillment  of  all  criteria  (similar  to  OER/EER) 


Knowledge  Factor  for  Weighting  -  Glide  Slope'  Approach 


AcquitMon  Progmtion  Factor  (Domain  Spaciftc) 

(Do  wa  h»va  tna  •yUam-apaoAc  MSI  hnoatoOga  that  wa  acOd  itauM  lor  tia  grran  acquuon  phaia?) 

Beh»nd  (•) 

The  goon  Wy  or  qual*?  o i  contrtwtng 
evidence  lc^*3-rt<3  to  T*  domjr  vccxe 

Itet  then  eipecWVwjurrt  lor  the 
current  eoqutvtioo  phete  Atocour** 
MMiwd  on  the  domen  ecore 

Concurrent  (1) 

The  quantty  and  quatiTy  of 
contrtumgendenceit 
corrmcrnurate  w«h  the  ecqiaeftoo 
Chaw  No  d*COun<  or  ncerove  * 

Ahead  (♦) 

The  quant* jr  and  QuaMy  of  contributing 
evidence  is  more  than 
«»pected  requ»red  lor  the  curent 
acQueflcn  phete  An  ncentne  lector  n 
aeeeeeed  on  the  doman  toore 

•  Centered  on  line  of  minimal  acceptability  as  zero 

•  Example: 

>  HFE  Score  of  2  (not  good)  with  AP  Factor  Behind  (-) 
nets  a  domain  score  of  1.8  points  out  of  5 
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Look  &  Feel  A 


i 


f 
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Look  &  Feel  C 


Coffl(Kthlivilw  HufMO  InttQfMion 
Ev*kj»t>on  Framewort  (CHIEF) 

ToMl  SyMem  Performance 

Implication 

Oomtn  R^og  AM 

|  W.wj 

tnNifK«m«n(  OplWMing 
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WORKSHOP  FOUR  PRESENTATION 


*  k 

Development  of  a  Comprehensive  Human 
Integration  Evaluation  Framework  (CHIEF) 
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Workshop  Goals 

0  Gather  information  to  assess  selected  HSI 
domains  for  PRR 

0  Rate  HSI  domains  using  the  CHIEF  tool  and 
rating  scales  provided 

0  Discuss  usability  of  the  CHIEF  tool  &  potential 
refinements 
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management  level 


Example  -  Measuring  HFE 


WIPT 


Coocet— •rper 

man  rruo  ac*« 


Management  Indicators 

Aggregated  Measures 


Jser  cnaraaerstos  .j'  ':  ■  "V 
&  worv Context  «[  ' 

— —  ---=~T  —  ^ 


moton 


■  i  vso  ty 

contros 


teTperature  aspsays 

’  M'_S7D  6S2-E 

MILSTD  147 2-G  * 

\  NOSH 


System  Attributes  & 
Environmental  Factors 


Policies  &  Standards 
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Semi-final  Rating  Scales  &  Criteria 


•  Minimal  acceptability  at  rating  of  3 

•  Ratings  assigned  by  fulfillment  of  all  criteria  in  each  block 
(similar  to  OER/EER) 


Acquisition  Progression  Factor  -  Glide  Slope'  Approach 


Acquisition  Progression  Fee  tor  (Domain  Specif*) 

(Oo  wo  have  the  sy*tom-spoc4<  HSI  knowledge  that  wo  woutt  shoUd  tor  the  gvon  aoqutubon  phase?) 

MMH 

The  quantity  of  quality  of  cortnbuong 
evidence  leadng  lo  Bio  domeo  *core  <% 
ie»»  then  ovpoctedrequeed  ice  vie 
current  ecqweAon  pheve 

Concurrent  (•) 

The  quantity  and  quality  o 1 
rorqCAjtng  evidence  a 
commensurate  wm  the  acqwuoon 
phate 

Ahead  (e) 

The  quarety  and  quatty  o f  ccntntx/tng 
evidence  it  more  than 
'•Ceded. required  lor  the  current 
acquisition  phaee 

•  In  practice: 

>  We  don’t  know  as  much  as  we  should  =  AP  Factor  Behind  (-) 

>  We  are  on  track’  for  HSI  knowledge  =  AP  Factor  neutral 

>  We  know  more  than  we  would  expect  to  at  the  given  acqu  . 
phase  =  AP  Factor  Behind  (+) 
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ihg  scales  &  criteria 


Comprehensive  Human  Integration 
Evaluation  Framework  (CHIEF) 


Total  System  Performance  Implication 


Performance  Support  A 
Training 


I 

e 
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Produce  A 
Oeplov 
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Data  Collection: 


1)  What  rating  and  Rating  and  APF  did  you 
arrive  at  for  your  domain? 

2)  How  did  you  arrive  at  this  rating? 

3)  What  measurements  or  analysis  did  you 
rely  on? 

4)  How  easy  or  difficult  was  it  to  match  your 
measures  in  the  CHIEF  rating  scale? 
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APPENDIX  C  -  DESIRED  CHARACTERISTICS  FOR  CHIEF 

SOFTWARE 


•  Basic  Functionality:  CHIEF  software  will  provide  a  web-based 
graphical  user  interface  and  database  for  collecting  and  assessing 
HSI  information  during  system  acquisition.  Users  will  be  able  to 
create,  edit,  approve  and  track  HSI  performance  ratings  and 
supporting  information  for  federal  acquisition  programs,  based  on 
the  CHIEF. 

•  Expected  User  Group:  The  CHIEF  user  group  will  include  HSI 
practitioners,  HSI  managers  and  other  acquisition  professionals 
from  a  variety  of  disciplines.  Users  will  include  federal  government 
employees,  military  members,  military  contractors  and  vendors. 

•  Interface  and  Display  Modes:  Information  entered  into  CHIEF  will 
serve  different  managerial  levels,  across  multiple  functional  groups. 
CHIEF  software  will  include  specific  interface  and  display  modes  to 
suit  the  needs  of  these  user  sub-groups.  CHIEF  information  (e.g., 
HSI  domain  definitions,  rating  scale  criteria,  rating  supporting 
information)  will  be  rapidly  accessible  as  “display-on-demand” 
information  using  hyperlinks,  mouse-over  or  similar  software 
functionality. 

•  Information  Input:  HSI  practitioners  will  carry  out  the  bulk  of  CHIEF 
data  entry,  edits,  and  updates.  File  formats  are  expected  to  include 
text  files,  spreadsheets,  photographs,  video,  scans,  and  electronic 
presentations.  Additional  formats  may  be  required. 

•  Information  Validation:  HSI  managers  will  validate  individual  HSI 
domain  ratings  and  supporting  information  entered  by  HSI 
practitioners.  Acquisition  managers  will  access,  review  and  approve 
top  level  information. 

•  Information  Export:  Reports  and  summary  information  from  CHIEF 
software  must  be  exportable  in  common  formats  including 
commercially-available  word  processing,  spreadsheet,  presentation 
and  document  display  software. 

•  Workflow  Automation:  Data  entry  will  be  implemented  according  to 
the  standardized  CHIEF  workflow,  using  government  approved 
reference  materials  and  standards.  The  software  should  automate 
the  workflow  through  the  use  of  pre-positioned  information,  drop¬ 
down  menus,  links  or  similar  functionality. 
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•  Access  Control:  An  access  hierarchy  will  be  required  to  manage 
each  user’s  ability  to  view,  edit  and  control  CHIEF  information. 
Access  to  CHIEF  software  will  be  controlled  by  the  appropriate 
authority  within  the  government  acquisitions  organization. 

•  IT  Enterprise  Compliance:  CHIEF  software  will  reside  on 
government  equipment  and  networks.  Users  and  administrators  will 
be  expected  to  have  access  to  the  Coast  Guard  data  network  or 
analogous  DOD  systems  (as  appropriate). 
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